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INTRODUCTION 


Introduction to the Practice Guide on Using 
Technology to Support Postsecondary Student Learning 


Despite increasing college enrollment! and growing 
diversity of the college student population,” college 
completion rates are low. Less than a quarter of 
students enrolled at public 2-year institutions 
complete their programs within 3 years.? At 4-year 
institutions, only 58 percent of students who enroll 
at public institutions and 69 percent of students 
who enroll at private institutions complete their 
programs at any institution within 6 years.’ The first 
year of college is critical, with about 20 percent of 
first-time full-time students in 4-year institutions and 
more than 40 percent of first-time full-time students 
in 2-year institutions failing to return to that same 
institution for their second year. Persistence and 
graduation rates are even lower for first-generation, 
ow-income, and racial/ethnic minority college 
students.° 


See the Glossary for a full list of key terms used 
in this guide and their definitions. These terms are 
underlined when first introduced in the guide. 


Many colleges are exploring ways to leverage 
technology to improve student retention and 
increase the educational options for and success of 
their diverse student bodies. Technology is infused 
in almost every aspect of college life. In the general 
population, 77% of individuals own a smartphone, 
73% own a laptop or personal computer, and 53% 
own a tablet.’ Mirroring that trend, students have 
increasingly greater access to personal computing 
and communications technologies.* Web-based 
course or learning management systems and 
instructional technology centers can be found on 
almost every U.S. college campus. Colleges are even 
ranked based on the quality of their technology 
infrastructure and connectivity.’ Off campus, 
technology allowed more than six million college 
students to take online courses in the 2015-2016 
school year.'° Colleges are using technology to 
improve the quality of student learning; make 


active and engaging learning available throughout 
institutional offerings; and help students become 
more successful learners. 


This practice guide, developed by the What 

Works Clearinghouse™ (WWC) in conjunction 

with an expert panel, focuses on promising 

uses of technologies associated with improving 
postsecondary student learning outcomes. It 
provides higher education instructors, instructional 


designers, administrators, and other staff with 
specific recommendations for supporting learning 
through the effective use of technology. 


Using Evidence to Develop the 
Recommendations 


This practice guide makes five evidence- 

based recommendations around how to use 
technology to support postsecondary learning. 
Each recommendation includes examples of 
technologies and how to implement them, advice 
on how to overcome potential obstacles, and a 
summary of the research evidence that supports 
the recommendation. The panel created a practice 
guide protocol to guide the evidence search and 
review.'' Findings from eligible studies (see Box 1) 
that meet evidence standards were summarized 
by trained and certified WWC reviewers for 
consideration by the expert panel. 


BOX 1. STUDY ELIGIBILITY CRITERIA 


Eligible research used a comparison group design, 
included an intervention that used technology to 
support student learning, involved college students 
in the United States, was published in 1997 or later, 
and reported on one or more outcomes in the 
following domains: (1) academic achievement; (2) 
college attendance; (3) credit accumulation and 
persistence; (4) attainment of a degree, certificate, or 
credential; (5) post-college employment and income; 
or (6) student engagement and motivation. 
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The expert panel, after considering the evidence, 
drafted the recommendations and assigned a level 
of evidence to each (see Box 2). 


Overarching Themes 


Each recommendation focuses on using technology 
to support particular aspects of student learning. 

Taken together, the recommendations highlight five 
BOX 2. LEVELS OF EVIDENCE themes that cut across all of the advice in this guide. 


Strong: There is consistent evidence that meets * Focus on how technology is used, not on the 


WWC standards and indicates that the practices 
improve student outcomes for a diverse population 
of students. 


Moderate: There is some evidence meeting WWC 
standards that the practices improve student 
outcomes, but there may be ambiguity about 
whether that improvement is the direct result of the 
practices or whether the findings can be replicated 
with a diverse population of students. 


Minimal: Evidence may not meet standards or may 
exhibit inconsistencies, but the panel determined 
that the recommendation must be included because 
the intervention is based on strong theory, is new 
and has not yet been studied, or is difficult to study 
with a rigorous research design. 


technology itself. Technology evolves rapidly. 
Many of the technologies used in the research 
that supports this guide could change in the near 
future. Thus the expert panel elected to focus on 
how technology can be used to enhance and 
support teaching and learning. Keep this in mind 
throughout the guide: the specific technologies 
are less important than the ways existing or 
emerging technologies can be used effectively in 
college settings both now and in the future. 


Technology should be aligned to specific 
learning goals. Every recommendation in 
this guide is based on one idea: finding ways 


to use technology to engage students and 
enhance their learning experiences. That is, the 


The recommendations and the panel's strength of 
focus of this guide is not on using technology 


evidence assessment are shown in Table 1 below. 


Table 1. Recommendations and corresponding levels of evidence 


LEVEL OF EVIDENCE 

Practice Recommendation Minimal Moderate Strong 
1, Use communication and collaboration tools to increase interaction 

among students and between students and instructors. e 
2. Use varied, personalized, and readily available digital resources to 

design and deliver instructional content. ® 
3. Incorporate technology that models and fosters self-regulated 

earning strategies. e 
4, Use technology to provide timely and targeted feedback on student 

performance. e 
5. Use simulation technologies that help students engage in complex 


problem-solving. 
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as a mechanical tool, for computation or word 
processing, for example. Instead, this guide is 
about using technology to engage students 
more deeply in learning content, activate their 
learning processes, and provide the social 
connections that are key to succeeding in college 
and in life after college. To do this effectively, any 
use of technology suggested in this guide must 
be aligned with learning goals or objectives. 
Instructors should explore the wide range of 
technologies available to help them meet their 
instructional goals. Then they should select 
technologies that best align with the pedagogical 
approach in their individual course(s). 


Pay attention to potential issues of 
accessibility. Despite the pervasiveness of 
technology and internet-enabled devices, 
technology is not higher education's panacea. 
Smartphone ownership is not universal, and 
some students in rural areas have spotty or 
nonexistent internet access. The issue of student 
access to technology has several facets. The 

Pew Research Center (2018a) has been tracking 
nternet access annually in the United States for 
the past 15 years. It finds that about 89 percent 
of all households have internet access. However, 
there are disparities based on age, income, race, 
region, and broadband capability. Generally 
older, minority, and immigrant populations have 
less access than younger, white households. 


Pew Research Center (2018a) reports broadband 
access, which is highly desirable for using and 
delivering media-rich material, is in approximately 
two-thirds of American households. This 
percentage is much lower among poorer 
populations and especially those in rural parts 
of the country. Thus colleges and universities 
with large minority commuter populations 

such as community colleges likely will have 
disproportionate numbers of students who have 
limited access to the internet in their homes. 


Students with special needs or disabilities can 
find it difficult to use or interact effectively with 
technology. For them, the issue of accessibility 
is critically important. Hundreds of colleges 


have been the subject of federal investigations 
regarding noncompliance with the Americans 
with Disabilities Act (Wang, 2017) because 

of limited access capabilities for students 

with special needs. Instructional designers 
increasingly must be aware of the service 
requirements for that student population. They 
need to ensure that learning materials on course 
websites and course/learning management 
systems can accommodate students who are 
visually and/or hearing impaired. 


Technology deployments may require 
significant investment and coordination. 
Implementing any new intervention takes 
training and support. That is true even more 

So in an environment where students might 

be more comfortable with technology than 

are their instructors and where resources 

are scarce. Effective implementation of new 
technologies that have the potential to support 
student learning outcomes requires support 
from administrators and teaching and learning 
centers. Further, sufficient information technology 
investments and infrastructure are necessary to 
roll out new technologies. Institutions should 
provide clear incentives and opportunities 

for professional development to encourage 
successful implementation. Finally, a commitment 
to collaboration among technology developers, 
instructional designers, higher education 
administrators, and instructors can promote 
effective implementation of new technologies. 


nstructors at smaller institutions that may 

ack established teaching and learning centers 
can obtain technology-related materials and 
instructional strategies from widely available 
sources. Those sources include Multimedia 
Educational Resource for Learning and Online 
Teaching (MERLOT), Creative Commons, the 
Online Learning Consortium, and EDUCAUSE, all 
described later in this guide. Instructors at any 
institution can benefit from having access to 
appropriate resources on campus to help them 
navigate issues such as software licensing and 
accessibility for all students. 
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e Rigorous research is limited and more is 
needed. Though the panel believes technology 
can be used to improve postsecondary student 
learning, the recommendations made in this 
practice guide are supported by moderate 
or minimal evidence. Despite technology's 
ubiquity in college settings, rigorous research 
on the effects of technological interventions on 
student outcomes is rather limited. More rigorous 
research on new technologies and how best to 
support instructors’ and administrators’ uses of 
technology is needed. 


Who Might Find This Guide Useful? 


This guide is designed to be used by instructors, 
administrators, and advisors at 2- and 4-year colleges 
and universities seeking ways to support instructional 
practices and student learning with technology. 

The recommendations in this guide will also be 
useful for technology developers, instructional 
designers, and staff at campus-based teaching and 
learning centers. Strategies for instructional design, 
application development, and deployment must 

all be considered to implement technology-based 
practices successfully in educational settings. This 
statement holds whether the technology is face-to- 
face, blended, or online. Where possible, this guide's 
suggestions for how to carry out a recommendation 
address all those audiences. 


How to Use This Practice Guide 


For each of the five recommendations in this guide, 
we include the following: 


e Recommendation: Details about the 
recommended practice, example technologies, 
Level of Evidence, and a description of 
how the recommended practice supports 
student learning. Appendix C contains a 
detailed rationale for the Level of Evidence 
and information on the individual studies 


that support each recommendation. Each 
recommendation also includes a “Spotlight 
on Implementation” that provides additional 
implementation detail from one of the 
studies that supports the recommendation. 
All figures and examples in this guide are 
offered as examples and should not be 

read as endorsements of specific products 
or approaches. 


¢ How to Carry Out the Recommendation: 
Guidance on how to implement the 
recommended practice. This guidance is informed 
by the studies that support the recommendation, 
as well as the expert panel's expertise and 
knowledge of resources and strategies available 
to help implement the recommendation. 


e Potential Obstacles: Advice from the panel about 
challenges and how to overcome them. 


e Tools and Resources: Examples related 
to the guidance for how to carry out the 
recommendation. 


The guidance on how to carry out each 
recommendation is presented in a sequence. 
However, not every step will be appropriate to every 
audience, nor is every step required in order to 
implement a recommendation successfully. Users 
of this guide are encouraged to use the advice 
provided here in ways that fit best into the varied 
contexts in which they work. Recommendations 1 
and 2 focus on interventions that might be applied 
broadly across a postsecondary institution, though 
they can also be implemented in individual courses. 
Recommendations 3, 4, and 5 focus on pedagogical 
strategies for using technologies in postsecondary 
courses, whether face-to-face, blended, or online. 


Though this guide does not provide step-by-step 
instructions for implementation, readers will find 
resources mentioned throughout providing more 
details about how to apply particular practices. 
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Recommendation 1: Use communication and collaboration 
tools to increase interaction among students and between 


students and instructors. 


When used effectively, communication and 
collaboration tools can increase engagement by 
allowing students to communicate about course 
content and their learning experiences. The 
relationships that college students form with their 
instructors and with one another enable them to 
create connections around common learning goals, 
build knowledge and identity, and develop a sense 
of belonging.'* These relationships and interactions 
are believed to foster student engagement and 
belonging, which may influence postsecondary 
students’ performance and persistence." 


Communication and collaboration tools can help 
build a sense of community and foster social 
learning. Social learning occurs when people learn 
from one another via observation, imitation, and 
modeling, and it can lead to established groups 
founded around common learning goals. Effective 
social learning in online courses is associated with 
better motivation and learning outcomes." 


Instructors and administrators in postsecondary 
settings can facilitate communication and 
collaboration among students and between 
students and instructors through the use of 
technology both inside and outside the classroom, 
whether those classrooms are traditional, blended, 
or exclusively online. 


The vast majority of students have access to 
affordable personal devices—including mobile 
phones, tablets, and laptops. They give students 
ready access to tools that can connect them with 


their peers both virtually and in person, encourage 
cooperation, permit continuation of discussions 
outside of class, and open up new avenues for 
engaging with instructors. Instructors should, 
however, be mindful of potential equity gaps. 
They need to think creatively about how they are 
maximizing learning opportunities for all students, 
including those who may not have access to 
personal devices. 


WWC staff and the expert panel assigned a minimal 
level of evidence based on three studies of the 
effectiveness of communication tools designed to 
foster collaboration and build community. One of the 
studies meets WWC group design standards without 
reservations,'° and the two other studies do not meet 
WWC group design standards.'® See Appendix C 

for a detailed rationale for the Level of Evidence for 
Recommendation 1. 


This section describes strategies, examples, and 
tools that can support instructors in effectively 
using communication and collaboration tools in 
their courses. 


How to Carry Out the 
Recommendation 


The guidance below is informed by the studies 
that support the recommendation, as well as 
the expert panel's expertise and knowledge 

of resources and strategies available to help 
implement the recommendation. 
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1. Select communication and collaboration tools 
that will best support learning objectives. 


When selecting technologies to support 


communication and collaboration, instructors 


should consider which tools are best suited 
to help students meet the objectives for the 


course or for individual activities within the 


course. Among several types of technologies 
to choose from, instructors might find it helpful 
to use one or more of the following types of 
communication and collaboration tools: 


ASYNCHRONOUS SYNCHRONOUS 
The tool Interact in a time- Interact in real or same 
allows users independent fashion time, and requires all 
UWOssa without participants participants to be present 
needing to be present at the same time, either 
at the same time. Can face-to-face or virtually. 
promote collaboration 
and facilitate group work if 
tool has features that allow 
for commenting on and 
creating and revising of 
multiple drafts. 
The tool can For one-to-one, one-to- To increase communication 
be used... many, or many-to-many between students and 
communications. instructors when they are 
not physically present in 
the same location. 
Can also facilitate remote 
collaboration among 
students assigned to work 
in small groups. 
Examples Email, online discussion LiveBoard, Google 
forums, blogs, wikis, Google Hangouts, and Zoom. Tools 
Docs, and microblogs such that allow for live or virtual 
as Twitter or Instagram. sessions in which users can 
interact from afar, such as 
screen sharing, tele- and 
video-conferencing, shared 
remote whiteboards, 
broadcasts of live 
simulations, and immersive, 
virtual worlds, 
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FEN STUDENTS AND INS 


SOCIAL NETWORKING 


Form communities or 
groups around similar 
interests or goals, and share 
content both within and 
outside of their groups. 


To structure social 
interaction among users. 
Communication with 
these tools can take 
place synchronously or 
asynchronously. 


Schools App, Course 
Networking, Commons In 
A Box (CBOX), Facebook, 
and Twitter. Most learning 
management systems 
also include features that 
allow social media-—like 
communication and 
collaboration tools to be 
embedded. 


MUNICATION 6 


TRUCTORS 


RECOMMENDATION 1 


For example, Schools App (see Figure 1.1) is a social Instructors should ensure that the use of these 
networking tool that allows students on a particular tools is not distracting students from their learning 
campus to collaborate and communicate during goals. Instructors should also check whether 
college. Features include a study group finder, any institutional policies restrict the use of social 
group and private messaging, and announcements networking tools in the classroom. 


from the institution. 


FIGURE 1.1. FEATURES OF A SAMPLE MOBILE APP THAT SUPPORTS COMMUNICATION 
AND COLLABORATION: SCHOOLS APP 


ENGAGE FORM A STUDY GROUP CONNECT 
Share your questions/ideas and Dont study alone! Connect with Chat with peers & staff privately 
see what your peers are up to fellow students and ace the test! or via group messages 


TargetX University Q Study Group Finder 


a Will Getin 
© Barbara Norayanansky 
ek an Wednesday Biology101 
study for 
iz. Coftee provid 


Chem 038 


ights 
he weekly quiz. Coffee 

Michelle Solano, Megan 
Biology)01 


Weill meet on Thursday at 5:30pm in the 
Library: Floor 2 to c&scues thet week's S Dennis Moloney 


notes. 


= Megan Sneeringer 
Q Scott Nelson 


SOURCE: https://www.targetx.com/solutions/schools-app/ 
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2. Consider student preferences and levels of 
access when adopting new technology. 


Instructors should consider student preferences 
and experience when selecting tools to adopt 
in their courses. Key considerations to keep in 
mind include the following: 


e Students in general are frequent users of 

mobile phones, tablets, and laptops, but 
different groups of students might have 
different preferences and varying degrees of 
access to these devices. 


e Some students might prefer social media 
technologies such as Twitter to institutional 
learning management system technologies 
because they are more familiar with social 
media technologies. 


e Other students might be apprehensive 
about mixing their personal and academic 
lives online. 


At the beginning of the semester, instructors can 
survey students on their use of technologies. 
Survey responses can help instructors to 
incorporate the tools students are already 

using or to plan time for students to get used 

to any unfamiliar technologies selected for the 
course. Instructors should carefully consider 

the implications of using communication and 
collaboration applications that students already 
use for non-academic purposes. For example, 
will the distractions of Facebook or Twitter in 
class outweigh the benefits of using the tool to 
support learning? Lastly, when instructors tailor 
course instruction and materials, they should 
remember the technologies—whether hardware 
or software—should always support learning 
goals and objectives. 
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3. Articulate expectations for when and how 


students should use communication and 
collaboration tools, both inside and outside 
the classroom. 


When introducing communication and 
collaboration tools—even those familiar to 
students—instructors must be clear about 

the purpose of using these tools within their 
course. Instructors should explain that these 
tools are meant to connect formal and informal 
learning by extending communication beyond 
the classroom. And that they can also foster 
collaboration around similar learning goals 

and tasks. 


Instructors should design discussion tasks to be 
relevant and integrated with course material, not 
perfunctory exercises in student participation. 
Specifically, technology-based communication 
and collaboration spaces should be treated as 
important components of the course. Students 
will benefit from clear expectations about 
when and how they should use those spaces. 
When use of these tools factor into instructors’ 
assessments of student performance, grading 
criteria and rubrics should be shared with 
students at the beginning of the course. 


Instructors should also provide guidelines for 
interacting respectfully and protecting privacy 
in communication and collaboration spaces. 
See Figure 1.2 for a sample set of guidelines 
students can receive at the beginning of the 
course to ensure they are aware of the norms 
for engaging in respectful behavior and 
dialogue and for protecting student information 
and privacy. 
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FIGURE 1.2. SAMPLE STUDENT GUIDELINES FOR INTERACTING ONLINE AND 


PROTECTING PRIVACY 


Y Be mindful of what you post. Anything posted will be permanently recorded online, so consider the 


impact that your words will have on others. 


v Protect your own information. Carefully review your privacy settings and consider how these settings 
affect who can and cannot view your content. Make adjustments to your settings as needed, and revisit 


your settings frequently. 


Y Protect the information of others. Be mindful of any posted content that may reveal information about 
others. This can include students’names and more general information such as their majors, addresses, 


social activities, and religious/political affiliations. 


Y Respect the diversity of the student and faculty body. Remember that other students and faculty will 
have different viewpoints on religious, political, and social issues. Try to be as objective as possible and 


respectful when discussing these issues. 


VY Give credit to others for their ideas and work. Also, remember to appropriately reference any direct 
quotes used from other sources. Not doing so is considered plagiarism. 


v Notify your instructor if you see posts or content that you think is disrespectful or makes you feel 


uncomfortable in any way. 


Y Familiarize yourself with campus resources and policies on protecting student and faculty privacy 


and security. 


SOURCE: Adapted from SANS.org guidelines on securely using social media: https://www.sans.org/security-awareness-training/ 


resources/top-tips-securely-using-social-media 


4. Seek information about communication and 
collaboration technologies currently available 
on campus, and determine whether there is 
guidance and support for using them. 


Instructors should check with other instructors 
and staff at their institution for existing 
resources, such as teaching and learning centers. 
Instructors can ask instructional designers about 
the availability of technologies that support 
communication and collaboration, as well as 
which are most popular among students and 
other instructors. For example, instructors can 
request learning management system usage 
data to help determine whether they should use 
communication functions within the system or 
look for alternative tools. 


Instructional designers might know whether 
the university has resources available to acquire 
additional technology that may be of use to 


instructors and students. They should also be able 
to tell instructors about any available guidance 

or training related to the effective use of such 
technologies to support teaching and learning. 


5. Monitor student participation, and provide 


facilitation and feedback as needed. 


Communication and collaboration tools that 
enable discussion forums and blogging can be 
used to provide students the opportunity to 
write reflectively, discuss learning experiences, 
and react to the experiences of classmates. 
Instructors can model desired behaviors; then 
if students are not engaging or contributing 

as expected, instructors can redirect them or 
provide additional support. 


When monitored and supported by instructors, 
communication and collaboration tools can 
facilitate knowledge building and knowledge 
sharing and also be effective in supporting 
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active learning.'” In addition, such tools 
provide students with additional options 

to communicate with one another and the 
instructor, making them more likely to engage 
in conversations. Instructors should encourage 
contributions from all students and discourage 
negative exchanges, such as bullying or 
unhelpful criticism, when they occur. In the 
online course setting, instructors must be 
diligent in monitoring communication boards 
and following up with students. 


Assess what is working (and what is not). 


Instructors should assess whether the selected 
communication and collaboration tools are 
effectively supporting interaction among 


students and between students and instructors. 


Answers to the following questions give insight 
on the quantity and quality of interactions: 


e Who participates in the conversations, 
and are there any distinguishing 
characteristics between those who do 
and do not participate? 


e Is the instructor able to respond to students in 
a timely fashion? 


e Are students engaging in meaningful 
conversation on the communication boards? 


e Is teamwork improving—for example, as 
gauged by the quality of group work 
and projects? 


Instructors should continuously revisit the 
learning outcomes of their courses and ensure 
any collaborative activities and communications 
they use align with—and support—these 
outcomes. Instructors should tap into resources 
at their institutions for help staying abreast of 
new technologies that might be able to further 
facilitate communication and collaboration in 
their courses. 
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Potential Obstacles and the Instructors can also consult publically available 

! . resources, such as the examples featured in 
Panel's Advice Figures 1.3 and 1.4. The Tech Select Decision Aide, 
OBSTACLE: Instructors might not be familiar with the featured in Figure 1.3, can be used to guide the 
technologies that can effectively foster communication selection of instructional technologies, including 
and collaboration. communication and collaboration tools. 
PANEL'S ADVICE: Professional development EDUCAUSE, a nonprofit association whose mission 
programs at both the campus and department is to advance higher education through the use 
level should feature workshops and trainings of information technology, offers several types of 
on technologies, including those that enhance research and publications. Its“7 Things You Should 
communication and collaboration. Instructors Know About" series (see Figure 1.4) includes briefs 
can consult with teaching and learning staff and/ on communication and collaboration tools. 


or instructional design centers at their institution 
to get one-on-one guidance for selecting and 
implementing these tools in their courses. 


FIGURE 1.3. EXAMPLE OF A TOOL THAT SUPPORTS SELECTION OF INSTRUCTIONAL 
TECHNOLOGY: TECH SELECT DECISION AIDE 


Tech Select Decision Aide - Each technology, like any tool, has its advantages and disadvantages. Choosing 
the wrong technology can lead to frustration, unnecessary time expenditures, and can negatively impact the 
user experience. The Tech Select Decision Aide seeks to optimize the selection process so that instructional 
designers, trainers, faculty, and other educational professionals will be more prepared to meet the training 
needs of their end users. Tech Select is available as a mobile app or through the web-based Learning Asset 
Technology Integration Support Tool (LATIST). 


<& ~ 
ca ; 


Your Decision 
Summary 


Technology: 


Blogs can support the following 
instructional strategies: 


Select by learning 
objective Elaboration ; 


i a 


Instructional Strategy: 


i af 


Learning Objective/Thinking Level 


Select by learning 


SOURCE: http://cehdclass.gmu.edu/ndabbagh/resources/TechSelectApp6.2/TechSelectApp6.2/index.html 
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FIGURE 1.4. EDUCAUSE’S “7 THINGS 
YOU SHOULD KNOW ABOUT” SERIES 


EDUCAUSE has a collection of concise briefs 
that summarize emerging technologies and 
practices and answer the seven following 
questions: 


What is it? 

How does it work? 

Who's doing it? 

Why is it significant? 
What are the downsides? 


Where is it going? 


SoOooooeoe 


What are the implications 
for higher education? 


The briefs on Collaborative Learning Spaces, 
Video Communication, and Assessing Online 
Team-Based Learning might interest instructors 
exploring communication and collaboration 
tools to adopt in their courses. 


SOURCE: https://www.educause.edu/research-and- 
publications/7-things-you-should-know-about 


OBSTACLE: /nstructors might feel overwhelmed 

by managing the added workload of responding to 
communications with students, especially if they are 
using multiple technologies to support asynchronous 
and synchronous social interactions. 


PANEL'S ADVICE: Instructors should not feel that 
they need to respond to every student post in 
online discussions. However, it is worth taking a 
few minutes to highlight key contributions—both 
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constructive and problematic—and to model the 
difference between productive and less productive 
contributions. When possible, they can use self- 
assessment, peer assessment, and peer evaluation 
to alleviate their workload. They also can limit the 
number of posts and amount of text per post. In 
large courses, instructors can also ask teaching 
assistants to help with monitoring and responding 
to student communications. 


Assigning students as discussion facilitators is 
another strategy. The strategy is most effective if the 
instructor creates multiple discussion forums with 
fewer numbers of students each. Doing so makes 
the instructor a more efficient facilitator of student- 
to-student exchanges; it also encourages students to 
own and engage in the learning process.'® 


OBSTACLE: Student participation and engagement 
with communication and collaboration tools is limited. 


PANEL'S ADVICE: Students may be resistant to the 
use of communication and collaboration platforms 
that instructors put in place for course activities 
and assignments. This resistance can stem from a 
reluctance to use new and unfamiliar tools and/or 

a dissatisfaction with the interface of the tools. The 
latter can be addressed by structuring asynchronous 
communication and requiring participation 
through the use of evaluation rubrics and instructor 
guidelines; guidelines can also be developed for 
active participation when using synchronous 
communication and collaboration tools. 


If students are reluctant to take up new 
technologies, instructors can leverage the tools 
that students are already using or are familiar with. 
This can increase the likelihood that they may be 
more comfortable and willing to use those tools to 
communicate with other students and instructors. 
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OBSTACLE: Privacy concerns can discourage concerns before adopting such tools for their 
instructors and students from using communication courses. Both instructors and students should be 
and collaboration tools. aware of and use strategies to protect themselves 


online. Sample privacy protection guidelines are 
featured in Figure 1.2 (above). Instructors should be 
as transparent as possible about their instructional 
use of communication and collaboration tools, 
particularly social media, and allow concerned 
students to opt out of using them. 


PANEL'S ADVICE: Sharing information online via 
communication and collaboration tools can pose 
some risk to the privacy of both students and 
instructors. That risk understandably might make 
some students and/or instructors uncomfortable, 
and instructors should carefully consider these 
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Recommendation 2: Use varied, personalized, and 
readily available digital resources to design and deliver 


instructional content. 


With instructor support and appropriate pedagogy, 
technology can help create productive educational 
experiences for larger numbers of students and 

a more diverse student population.'? Traditional 
course formats and materials, on the other hand, 
can limit access to educational opportunities. 
Approaches to teaching and learning that require 
students to be in a specific place, at a specific 

time, and for a fixed duration necessarily bound 
opportunities for interaction. Those boundaries can 
limit the ways in which students work with both 
faculty and peers to solve problems, seek help, 

or engage deeply with content in and out of the 
classroom. Similarly, familiar educational resources, 
such as textbooks and readings, limit how content 
can be presented to students and how they can 
process that content. 


The panel recommends that institutions of higher 
education leverage technology to help students 
learn more productively by: 


e (Recommendation 2a) varying, blending, or 
accelerating course formats; and 


e (Recommendation 2b) packaging course 
content to minimize cost, maximize accessibility, 


and accommodate different learning preferences. 


Recommendation 2a focuses on the format of 

an entire course. Specifically, administrators can 
encourage instructors and instructional designers 
to use technology to create blended or flipped 
courses, in which some lecture material and other 


course content are delivered online. These formats 
may preserve more class time for hands-on and 
other experiential activities, group assignments, 
and individualized instruction, rather than for 
lecture. Such formats can also be used to accelerate 
instruction; for example, by allowing students to 
accelerate their completion of a course of study or 
to transition more quickly from developmental to 
credit-bearing courses. 


Recommendation 2b focuses on interventions 

that can be applied at either the course or the 
module level. Technology can be used to package 
content in multiple ways that help students access 
and study course materials. This is especially true 
when the interface is interactive, flexible, and offers 
multiple ways and times for students to access the 
content. Students’ understanding of course content 
can be deepened by providing them with digital 
representations of that content that help them 
visualize complex processes. Course interfaces can 
also include digital review and study tools, such as 
podcasts, that provide multiple ways for students to 
access and study course material. 


WWC staff and the panel assigned a moderate 
level of evidence based on 16 studies. Eleven 
studies meet WWC group design standards 
without reservations.”°*' Five studies meet WWC 
group design standards with reservations.” See 
Appendix C for a detailed rationale for the Level of 
Evidence for Recommendation 2. 
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SPOTLIGHT ON IMPLEMENTATION: ACCELERATED BLENDED LEARNING 


The Open Learning Initiative (OLI) project at Carnegie Mellon University provides free, online 

course content to students in a variety of subjects. OLI is an open education resource that can be used to 
structure courses in an accelerated, blended format rather than a traditional, face-to-face format. The OLI- 
Statistics course at Carnegie Mellon University covers introductory statistics topics, allows students to practice 
applications of those topics, and provides immediate and targeted feedback on understanding of topics. The 
8-week OLI-Statistics allows students to accelerate course completion, while the traditional face-to-face version 
of the course takes 15 weeks to complete. Students in the OL| course also meet with an instructor twice a week 
in 50-minute sessions to ask questions and review challenging material. 


The OLI-Statistics course provides most of its instructional content online. Course topics are presented in a 
hierarchy, visible from a navigation bar on the left side of the webpage. Instructional content includes short 
visual animations of statistical concepts with narration. At key transitions between topics, the system reminds 
students of how statistical concepts connect with one another. The system provides multiple opportunities for 
practice throughout each lesson, including: 


Comprehension questions; 
Real-life situations to test understanding; 
Short-answer reflection questions; and 


A tool to experiment with statistical properties such as the mean and median. 
“Mini-tutors” give students real-time hints and feedback as they practice their skills. 


Study Findings: Lovett and colleagues (2008) report the impact of the flipped classroom format on the 
comprehensive assessment in statistics course measure was positive and statistically significant. 


SPOTLIGHT ON IMPLEMENTATION: DIGITAL VISUALIZATION THROUGH TIME 
COMPRESSED ANIMATED DELIVERY 


Time Compressed Animated Delivery (TCAD) is a method of presenting information using two- and three- 
dimensional animations. This method is particularly well suited for teaching students about complex processes 
covered in science courses. TCAD was used by Trevisan, Oki, and Senger (2010) to deliver an animated digital 
representation of follicular dynamics, a key process in reproductive physiology that often confuses students 
because it is a cyclic change in anatomy that is integrated with other complex physiological processes. 
Trevisan and colleagues used a 17-minute TCAD video on follicular dynamics in an undergraduate reproductive 
physiology lecture. They followed these steps to develop the TCAD video: 


. Storyboarding to visualize the instructional design; 


. Producing step-animations by starting with static images that can then be animated according to 
storyboard specifications and synchronized with voice-over descriptions of the topic; and 


. Developing 3-D anatomical models with the appropriate texture, transparency, and lighting, which are 
then animated. 


Study Findings: The impact of the animated presentation on students’academic achievement was positive 
and statistically significant. 
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How to Carry Out the 
Recommendation 


The guidance below is informed by the studies 
that support the recommendation, as well as 
the expert panel's expertise and knowledge of 
resources and strategies available to help 
implement the recommendation. 


Applies to Recommendation 2a 


1. Leverage technology to vary the format for 
delivering courses. 


A variety of options exist to structure a course 

in ways that differ from traditional classroom 
instruction. Blended learning formats, which 
provide a combination of online and classroom 
instruction, can leverage the advantages of each 
instructional delivery approach. This provides 


students with opportunities both to learn at their 


own pace and on their own time and also to 
interact with an instructor and peers online and 
in person during face-to-face classes. 


Flipped classroom models used in blended 
settings provide lecture material outside of 
class time, often through videos and podcasts, 
allowing students to learn and process lesson 
content at their own pace. This frees up class 
time for hands-on activities, group assignments, 
and learning experiences tailored to the 
preferences and interests of different students. 


Interactive online course modules provide 
asynchronous online instruction that can be 
structured to provide students the benefit of 
immediate feedback through adaptive learning 


features. As appropriate, instructors can also use 
technology to encourage self-paced learning or 
to accelerate instruction, allowing students to 
speed the completion of a course of study or to 
catch up with the rest of their cohort.” 


Instructional designers may be more aware of 
information about the effectiveness of available 
technologies than are instructors. Designers can 
collaborate with instructors to help them select 
course delivery structures that are appropriate 
for the content of the course and the students 
taking the course. They also can help instructors 
plan and implement blended and flipped course 
formats. Further, they can guide instructors in their 
selection of online course activities, including 
interactive modules that provide students 
feedback as they navigate the lesson content. 


The ADDIE (Analyze, Design, Develop, 
mplement, Evaluate) model offers a framework 
for designing and developing education and 
training programs. Instructors and instructional 
designers can use ADDIE to develop flexible 
course delivery formats—and appropriate 
content and materials—that align with course 
goals and objectives. The ADDIE model, a five- 
phase course-development process created in 
the 1970s by Florida State University for the U.S. 
military, is featured in Figure 2.1. 


There are several considerations in planning 
for online education, including those related 
to institutional context, available finances, and 
hardware and software. Key considerations 
for planning online instruction (adapted from 
Picciano, 2015) are summarized in Figure 2.2. 
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FIGURE 2.1. THE ADDIE MODEL: A SAMPLE FRAMEWORK FOR DEVELOPING COURSES 


ANALYSIS 


* Conducting a needs analysis 

» Identifying the knowledge gap 

* Conducting an audience analysis 
> Developing learning objectives 


| DESIGN 


- Identifying instructional design strategy 
- Selecting appropriate delivery method 

- Developing storyboards and media 

- Establishing an evaluation methodology 


EVALUATION 


* Collecting evaluation data 


- Assessing project performance 
- Reporting results 


IMPLEMENTATION DEVELOPMENT 
» Establishing schedule - Creating the prototype 
» Preparing training materials - Developing training materials 
» Notifying and enrolling learners - Completing a tabletop review 


» Launching the course 


SOURCE: http://www.elearning.niu.edu/services/onlinecoursedev.shtml 
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FIGURE 2.2. SAMPLE CHECKLIST FOR PLANNING ONLINE EDUCATION 


PLANNING FOR ONLINE EDUCATION 


Y Environmental Scanning — An external review of the educational environment is needed to ensure 
that technology is appropriate, does not become obsolete quickly, and addresses key considerations 
in higher education, including (a) enrollment patterns such as expanding enrollment and expansion of 
life-long learning, (6) student retention, (©) rising tuition and student loan debt, (d) government 
oversight and accountability, and (e) commoditization of higher education. 


v Developing Goals — Institutional, educational, and social goals of an institution should be considered. 
For example, goals for online education may include an expansion of social life, attainment of more 
research grants, or increasing an institution's research profile. No two campuses have the same goals. 


Y Online Education Applications — In addition to fully online, massive open online course (MOOC), 
and blended learning applications, online education applications can include (a) adding learning 
analytics software to monitor student progress, (b) making greater use of social media software in 
existing courses, (c) integrating mobile technologies in instruction, (d) creating genres of courses with 
blended or flipped classrooms, (e) adding adaptive learning software to learning environments, and 
(f) undertaking blended learning initiatives in areas such as STEM that have more specific or complex 
learning needs. 


VY Hardware - Hardware needs are generally straightforward, and must support a network size 
appropriate for the application. Additional considerations include (a) use of mobile devices instead of 
laptops in classrooms and (b) supporting faculty acquisition of laptops. 


VY Software - Most colleges and universities use a learning management system (LMS). Regardless of 
which LMS a college chooses, substantial in-house programming and systems support is needed. An 
LMS should be integrated with a college's database to allow for the efficient transfer of course and 
student data. Third-party vendors may also be contracted to develop online courses and MOOCs. 


Y  Staff/Faculty Development - Faculty “buy in’ is critical to the success of any technology application. 
By involving staff and faculty in the planning and decision-making process, colleges can roll out 
technology-enabled courses and course enhancements more effectively. Staff training and support (e.g., 
one-on-one coaching, incentives) can facilitate the take-up of new technologies. 


Y Facilities and Infrastructure - Central tech nology supports such as campus-wide wi-fi and help desk 
support are needed to successfully implement online courses, as is instructional design support to help 
instructors transition to new course delivery modalities. 
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PLANNING FOR ONLINE EDUCATION CONTINUED 


Y Finances — Online education initiatives require funding, which can be substantial with bold new 
initiatives or relatively modest with incremental enhancements. While online education may expand 
course enrollment, results are mixed as to whether these technologies can improve a college's financial 
position. Finances must also be carefully monitored and a contingency plan made when working with 
third-party vendors, who may overpromise and under-deliver on their products. 


Y Policies - Existing policies should be reviewed to ensure that those related to curriculum approval, 
workload, intellectual property, accreditation compliance, faculty observations, and evaluations are 
not infringed upon. Colleges should also be aware of external compliance issues, such as Section 504 
of the Rehabilitation Act of 1973 and the Americans with Disabilities Act (ADA) of 1990. Although 
most major learning management systems are accessible to students with disabilities, this should be 
monitored. Moreover, copyright infringement laws should be reviewed before developing or expanding 
online education. 


Y Evaluation - Evaluation criteria should be developed that relate to the goals and objectives of the 
online education application. The data capture capabilities of online and blended learning have opened 
up new areas of evaluation. For example, instructional transactions on blogs, discussion boards, and 
wikis can be measured and monitored. Faculty experiences with new technologies should also be 
monitored, and continued long after initial implementation. 


SOURCE: Adapted from Picciano, A.G. (2015). Planning for Online Education: A Systems Model. Online Learning Journal, 19(5), 142-158. 
https://onlinelearningconsortium.org/jaln_full_issue/online-learning-journal-volume-19-issue-5-december-2015/ 


Applies to Recommendation 2b Instructors must also consider their method(s) 
for delivering content. Very complex material 
2. Plan instruction so that course content is being introduced for the first time might be 
carefully packaged and sequenced, and use effectively delivered via an audio podcast 


with accompanying visuals, as opposed to an 
audio-only podcast, for example. Instructors 

can accommodate students’ varied learning 
preferences by offering multiple pathways to 
learning the same material. One way to do 

this is to package content in more than one 
format. Instructors can provide content in both 
video and print formats, giving learners some 
control over the media they use to consume the 
information. When students are learning through 
online modules, instructor-led sequencing of the 
information presented can serve as a scaffold to 
support student learning.” 


technology to scaffold students’ acquisition 
and application of the content. 


nstructors who wish to adopt blended or flipped 
course formats should ensure the course content 
is suitably matched to the selected course 
structure and delivery modalities. In doing so, 
instructors decide which content will be covered 
in online modules, in class, or in both learning 
environments. Instructors who are accelerating a 
course using a restructured format should ensure 
that the material of that course is conducive to 

a condensed schedule. All instructors should 
also consider how to extend learning by using 
communication and collaboration tools to 
prompt or continue discussions outside of class 
sessions (see Recommendation 1). 
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With support from their colleagues or 
instructional designers, instructors should aim to 


Applies to Recommendations 2a 
and 2b 


incorporate multiple media formats throughout 
the duration of a course. By presenting material 
in multiple ways, instructors may help students 
develop mental representations, reduce their 
cognitive load, and test their understanding 

of content. Students’ understanding of course 
material may be deepened through engaging 

in interactive online modules or by viewing 
simulations or visualizing content in new ways. 
Technology such as podcasts, instructional 
videos, and other online media may be used to 
maximize efficiencies in students’ review of course 
material and homework. Students can also use 
technologies to demonstrate their understanding 
and apply their knowledge. 


Tutorials to help instructors learn about 
instructional design principles for packaging, 
sequencing, and delivering course material 

via technology can supplement on-campus 
professional development on course delivery 
technologies. Course content can often be 
embedded in learning management system 
platforms. Instructors also can guide students to 
access content from Open Educational Resources, 
including resources available from both internet 
and intranet sites. 


3. When varying course formats or course 


packaging, instructors should clearly 
communicate expectations for what students 
should do during the course. 


In addition to describing a course's goals and 
objectives, instructors must explain its format and 
modes for delivering its content. They also should 
describe their expectations for what students 
need to be prepared to do before, during, and 
after class sessions. For example, if the instructor 
is adopting a flipped classroom model, students 
need to understand the amount of study and 
preparation that will be required of them before 
and during those face-to-face sessions. 


Instructors might plan to primarily follow 

a traditional face-to-face format but also 
incorporate various technologies to deliver or 
review course content. They should demonstrate 
to students how they are expected to use 
those technologies to support their learning or 
complete assignments. For example, instructors 
who incorporate content from an adaptive 
learning course should explain not only how 
students access its learning modules, but also 
how students’ completion of—and feedback 
from—those modules connects to other course 
activities and assignments. 
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Instructors should develop and share grading 
rubrics with students so they know what 

they must do to demonstrate successful class 
preparation, participation, and reflection in both 
online and face-to-face learning environments. 


4. Monitor and evaluate the effectiveness of 
course formats and use of multimedia to 
deliver content. 


Instructors, department heads, and 


formats and delivery modes used to engage and 
teach learners. Together, they can determine 
which instructional approaches should be 
continued—or discontinued—with future 
student cohorts. The Practical Evaluation for 
Digital Learning (PEDL) toolkit offers guides on 
how each of these stakeholders can evaluate 
digital learning interventions.”° The PEDL process 
and examples of tools included in the PEDL 


toolkit are shown 


in Figure 2.3. 


administrators share responsibility in monitoring 
and evaluating the effectiveness of the course 


FIGURE 2.3. SAMPLE TOOLKIT FOR EVALUATING DIGITAL LEARNING INTERVENTIONS: 
PRACTICAL EVALUATION FOR DIGITAL LEARNING (PEDL) 


Tool 3a: Evaluating Small Course/Single Course 
section if you are already using the technology 
(focuses mainly on identifying needs, obtaining 
data, collecting qualitative feedback, and assessing 
further information needs). 


Practical Evaluation for Digital Learning (PEDL) 


Welcome 


valu 


he vet 
Getting Started to star at 


, . ee Tool 6: Checklist of Roles (describes types of 
ee staff members on campus who can help with 
each part of the study). 


Tool 5: Study Design Options (choosing the 
right options based on the decision that 
needs to be made; brief overview of RCT, 
QED, and descriptive study designs). 


SOURCE: http://evaltoolkit.wpengine.com/ Username: demo Password: evaluation 
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Potential Obstacles and the 
Panel’s Advice 


OBSTACLE: /nstructors might find it difficult to design 
or adapt course materials that apply new technologies, 
especially if administrators are unable to provide 
resources to support such efforts. 


PANEL'S ADVICE: It is essential that good design 
principles be applied prior to restructuring a 
course to use technology. Administrators can 
lead collaborative design efforts. Such efforts 
might include working with an instructional 
designer to develop or adapt a range of tools for 
instructors to use, and then training instructors 
on how to use those tools effectively. As available, 
instructors can be supported by a campus's 
center for teaching and learning staff, who can 
provide guidance on how to apply good design 
principles to technology-supported course 
formats. Instructional designers can also share 
their knowledge of which content creation and 
delivery technologies are popular amongst other 
instructors. Designers also can share any insight 
they have about which multimedia formats align 
with students’ varying learning preferences. 


When developing courses that include online 
lessons or activities, instructors can borrow or 

adapt existing materials if they lack the time 

and resources to develop new materials. With 

the help of instructional designers, instructors 

can integrate into their courses openly available 
content, such as examples from YouTube, Creative 
Commons, MERLOT, or SkillsCommons. An overview 
of the resources available in MERLOT is provided in 
Figure 2.4. 


The MERLOT collection consists of tens of thousands 
of discipline-specific learning materials, learning 
exercises, and Content Builder webpages. All 

of these items have been contributed by the 


MERLOT member community. Some contributors 
have authored the materials themselves. Other 
contributors have discovered the materials, found 
them useful, and wished to share their enthusiasm 
for the materials with others in the teaching 

and learning community. MERLOT also offers 

a customized search engine for educators that 
searches more than 60 other free and open online 
libraries of teaching materials as well as the web. 


Instructors who want the ability to personalize 
course material can focus on resources and tools 
that allow them to customize content delivery. Some 
Open Educational Resources materials can be used 
to customize content, instead of instructors building 
content from scratch. 


OBSTACLE: Students’ focus on the content of a lesson 
could be compromised if they are preoccupied learning 
how to use new technologies. 


PANEL'S ADVICE: Students may be unfamiliar with 
the latest technologies. Instructors should consider 
leveraging technology that students have already 
used or that students will find intuitive. Instructors 
also should consider using technologies that are 
widely used on campus, as these might come 

with more help desk support either from campus 
instructional design or technology staff or from the 
developers of the platform or software. 


At the beginning of the semester and as needed 
throughout it, instructors should provide students 
with an orientation to each of the technologies 
they use, as well as information about where to 

get help if they encounter difficulties using it. 
When learning a new technology is a course 
objective, such as when students in a 
communications course are expected to know 
how to use social networking effectively, instructors 
should provide students with practice opportunities 
to develop fluency with the technology. 
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FIGURE 2.4. SAMPLE REPOSITORY OF OPEN EDUCATIONAL RESOURCES: MERLOT 
(MULTIMEDIA EDUCATIONAL RESOURCE FOR LEARNING AND ONLINE TEACHING) 


The MERLOT collection consists of tens of thousands of discipline-specific learning materials, learning 

exercises, and Content Builder webpages, together with associated comments, and bookmark collections, 

all intended to enhance the teaching experience of using a learning material. All of these items have been 
contributed by the MERLOT member community, who have either authored the materials themselves, or who 
have discovered the materials, found them useful, and wished to share their enthusiasm for the materials with 
others in the teaching and learning community. MERLOT also searches over 60 other free and open online libraries 
of teaching materials as well as the web with a customized search engine for educators. 


Browse the MERLOT 
collection by 
discipline, 
material type, 
audience, 
technical 
format, 
mobile 
platform, or 
other filters. 


Browse by Discipline MERLOT Collection 


All Disciplines > 


MERLOT Materials 


NA frem tre 


Results of the search 
appear as an easy- 


to-navigate grid Material Detail 
where the user can = : 


. Quality 
nd ‘arth Evolution 
easily identity the 1h snenpatonssn Sn 9 atin Ronkson espe lamarese ene sere aeite tas 
Aj exercises and simulations keyed to the episodes of the show. The Evolution Library hes a atny eee 
material type, author, large number of ssetul resources indexes by topic. While & led as sreterance ste, © pecoh aided ade 
. 7 n CONTAINS animations, SienZabons. and Lutorials as part of the larger site Learning, Euercises (2 
date created, date ois wr] | = ES aroun eer 
A rt ral a — a Science avd Technoleey / Agiouture and Environmental Sdences: yo okra 7 
modified, and peer tore 


G) menor cuss 


and user ratings. — | oto tane m Pewee 


Bi Goolmnark / Add to Course ePortfote on ec0ce 


GF Crosto a Learning Exorcso epert Broken Unk 
@ Ado Accesabiity infermation Add a Comrenest Report as imappeoprate 


More about this material 


Material Type: Aeference Material Mobile Compatibility: Not specified at this rene 

Date Added to MERLOT: jarwary 21, 2007 Technical Requirements: Some parts repire shocewarn anc ether foalviseo ar} 
Date Modified in MERLOT: Warct 16 2018 QuickTime 

Author: Acne Zeber and jes0n J. Hurke. WGHM Press Contact and Clear Sive Sky Language: Engjtsn 

Precuciors a Cost invalved: no 

Submisen: Marty Zatey Source Code Available: 0 

Primary Audience: Mickle School, High Schcol College General Ed Accessiblity Information Available: > 

Technical Format: Website Creative Commens: no 


Click on a resource to access summary information and the 
material itself, as well as quick reference information and a 
SOURCE: https://www.merlot.org/merlot/index.htm comments section to communicate directly with other users. 
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OBSTACLE: Because technology affords students the 
opportunity to learn more on their own time, it might 
be difficult to quickly identify students who are not 
learning effectively with new technologies. 


PANEL'S ADVICE: Progress monitoring is important 
regardless of whether a course is delivered online or 
in a traditional format. There are some technologies 
that can make it much easier for instructors 

to identify students who are at risk of poor 
performance. For example, instructors of students 
completing modules through the Open Learning 
Initiative have access to real-time feedback and data 
provided to students.” Instructors then can use 


these data to identify students in need of targeted 
interventions or extra support. 


A sample Open Learning Initiative “learning 
dashboard" available to instructors is featured 

in Figure 2.5. Instructors can also work with 
instructional designers to learn how to use 

earning management system data to track student 
engagement with course materials. Further, 
instructors should work with students to help them 
earn how to read and interpret their own activity 
data, including giving them benchmarks of how 
much time on task is associated with above average 
and excellent performance. 


FIGURE 2.5. SAMPLE OPEN LEARNING INITIATIVE (OLI) LEARNING DASHBOARD 


Data from student activities and assessments are displayed in OLI's Learning Dashboards, which are specific 
to each course. Each Learning Dashboard includes information about student participation and performance 
at both the class and individual-student level. OLI recommends reviewing its Learning Dashboard before 
each class session and allocating in-class time to either address difficult topics with which students are 
struggling or push forward with more challenging material if data suggest students are excelling with the 


previous concepts. 


Modale 2 
—! Examining Relationships 


SOURCE: http://oli.cmu.edu/get-to-know-oli/course-features/ 
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OBSTACLE: Some students might not be able to afford 
the devices such as smartphones or laptops needed 

to access education materials, a shortcoming often 
referred to as the ‘digital divide.” 


PANEL'S ADVICE: In designing and providing 
instructional material that students will use online, 
designers and instructors must understand who 
their students are and their access to technology. 
Many colleges and universities survey new students 
to determine the nature and extent of their access 
to internet technology for the purpose of informing 
campus investments. As a result, many colleges and 
universities provide expanded access to technology 
facilities on campus. Though a step in the right 
direction, campus access is not comparable to 
access at home where students can conveniently 
engage in course activities anytime day or night and 
as frequently as they wish. 


At the beginning of the semester, instructors 
should invite students to fill out a survey to report 
whether they currently use or have access to the 
technologies that will be used in the course. This 
helps instructors understand and plan for the needs 
of the overall class around access and fluency with 
the selected technologies. 


When feasible, instructors should select technology 
that students can access at low or no cost, or be 
prepared to help seek institutional support for 
students who need financial help to access it. Faculty 
should have conversations and explore options with 
administrators, the campuses Information Technology 
department, and relevant student support offices such 
as Financial Aid in advance of adopting technology 
whose costs may be passed on to students. Pedagogy 
First, featured in Figure 2.6, is an example of a free 
tool that supports technology selection. Users can 

use the tool’s screening function to quickly identify 
technologies that are free for instructors and students. 


FIGURE 2.6. SAMPLE TOOL TO GUIDE SELECTION OF TECHNOLOGY: PEDAGOGY FIRST 


The Pedagogy First tool created by the Grand Canyon University - Center for Innovation in Research 
and Teaching provides a decision tree to help instructors select a specific technology that meets their 
pedagogical needs. Users can search for specific technologies they are already familiar with, or they 
can answer a series of questions to narrow down the list of available technologies. 


Hewmar sy 


s@l?.°Pedago 
Db First’ “i 


cmon 
Maccierract, 2 Lom, H, Wilhere. (1.2 fects, 1 (2026). A 
sara abse ot even ety oq Ar coe 


Pedagogical Value 


What de you hope ts accomplish with the integration of techaology? 


Resources matching your selections: 74 


As users select 

what they hope to 
accomplish with 

the integration of 
technology, levels 

of student and 
instructor expertise 
with technology, 
and how much 

time the instructor 
can dedicate to 
initial setup, the 
number of resources 
matching the user's 
selections is updated 
and technology 
recommendations 
are provided. 


SOURCE: Mandernach, B. J., Love, H., Williams, B. & Brooks, J. (2016). Pedagogy First. Center for Innovation in Research and Teaching, Grand 
Canyon University. Available at: https://cirt.gcu.edu/pedagogyfirst 
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Recommendation 3: Incorporate technology that models 
and fosters self-regulated learning strategies. 


The panel recommends using technologies that 
facilitate the incorporation of self-requlated learning 
into all learning environments, including face-to- 
face, blended, and online course delivery formats, as 
a way to help students successfully self-direct their 
engagement with course content and learn more 
effectively.” Self-regulated learning is a process 
in which students understand and control their 
own learning. It is driven by meta-cognition, which 
involves thinking about one's thinking, strategic 
actions such as planning and self-monitoring, and 
self-efficacy and motivation to learn. 


Effective learning depends on knowing how 

to learn and to manage one's own learning 
processes. Some students gain these skills prior 

to enrolling in college, but many do not. Those 
who do not often struggle once in college, 

where instructor supervision can be minimal and 
students have considerable autonomy in their 
educational activities. They may lack awareness and 
understanding of their own learning processes and 
struggle to self-direct their learning activities. Online 
and blended learning environments can magnify 
these challenges because they can require extra 
levels of self-direction, organization, and planning 
from the student.®”? 


To address these challenges, technology can be 
used to activate or scaffold the following self- 
regulation skills:°° 


e Goal setting: Prompts can be embedded in 
online courses, or in text messages outside 
of a face-to-face class, to encourage students 
to set learning goals. For example, rubrics 
may accompany activities to help students 
understand how to successfully meet the 
expectations of the task at hand. 


e Strategies to organize, code, and rehearse 
information: Note-taking features can be 
incorporated into learning modules, or note- 
taking applications can be provided on a 
course learning management system. Concept 
mapping tools can be used to help students 
organize and comprehend large amounts of 
information. Reminders delivered via email, text 
message, or smartphone can prompt students 
to study or rehearse. Examples of these 
features can be found in the “Spotlight on 
Implementation” below. 


e Self-monitoring: Online interactive modules 
can be programmed to model ways to self- 
monitor by asking students to record reflections 
on their thought processes or justifications for 
their answers. Modules can also be programmed 
to help students keep track of their study time. 


e Self-evaluating and self-correcting: Web- 
based courses or modules can provide flexible 
ways for students to self-evaluate through 
embedded knowledge checks, or they can be 
programmed to provide clues for self-correcting. 
Embedded knowledge checks appear within the 
learning material. They can aid learning without 
increasing students’ cognitive load, while also 
providing students with immediate feedback on 
which content they have mastered or need to 
continue reviewing. 


e Help-seeking: Web-based modules or courses, 
course and learning management systems, 
communication and collaboration tools, content 
creation and delivery tools, social media, and 
the like, can all be used to foster help-seeking by 
providing easy access to both online resources 
and tutoring supports. 
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e Time planning and management: 

Myriad applications are available for use in 
web-based modules, course and learning 
management systems, and smartphones, such 


WWC staff and the panel assigned a moderate 
level of evidence based on four studies 

that examined the effectiveness of various 
strategies to foster or activate student self- 


as task reminders embedded in calendars or 
visualizations of timelines. 


regulation. All four studies meet WWC group 
design standards without reservations.*' See 
Appendix C for a detailed rationale for the 
Level of Evidence for Recommendation 3. 


SPOTLIGHT ON IMPLEMENTATION: NOTE TAKING AND SELF-MONITORING PROMPTS TO 
SUPPORT SELF-REGULATED LEARNING 


Note taking is a critical skill for postsecondary students. Taking good notes enables students to effectively 
encode information and refer back to it later. Kauffman and colleagues (2011) investigated how note- 
taking methods and self-monitoring prompts interact and influence students’ collection, organization, 

and integration of online information, as well as achievement. Students took notes on a web-based 

article on educational measurement. Students were assigned to use one of three note-taking methods— 
conventional, outline, or matrix. As they proceeded through the educational measurement article, some of 
the students received electronic self-monitoring prompts, and corresponding questions, like: 


“There was a lot of information covered on that webpage. Now would be a good time to ask yourself 
if you have collected all the important information. If you believe you can answer the question below, 
even with your notes, then you are probably ready to move on to the next section. Otherwise you may 
want to return to the Central Tendency and Dispersion page’ 


Study Findings: The impact of online note taking using the matrix method on student achievement was 
positive and statistically significant, while the outline method had indeterminate effects. 
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want students to accomplish with regard to self- 
regulated learning—either throughout a course 
or during a specific lesson or unit—and select 
tools to help students acquire and apply self- 
regulated learning strategies. 


How to Carry Out 
the Recommendation 


The guidance below is informed by the studies 
that support the recommendation, as well as the 
expert panel's expertise and knowledge of 
resources and strategies available to help 
implement the recommendation. 


Instructors should select tools and technologies 
with interactive interfaces that can be used to 
encourage and support students’ use of self- 


1. Select tools and technologies that serve regulated learning strategies while completing 


as scaffolds to help students apply self- 
regulated learning strategies. 


Some tools might help students learn specific 
content, such as a lesson within a unit, whereas 
others might support the way students engage 
with the course and its related content more 
generally. Instructors should consider what they 


activities in online learning environments. For 
example, video-based learning modules with 
interactive components, including note-taking 
features, practice questions, and supplemental 
resources related to the topic, are one way to 
boost student engagement and scaffold students’ 
use of self-regulated learning strategies while 


learning in video-based settings (see Figure 3.1).*? 


FIGURE 3.1. EXAMPLE OF INTERACTIVE INTERFACE TO SUPPORT STUDENTS’ USE OF 
SELF-REGULATED LEARNING STRATEGIES 


Interactive video-based interfaces allow students to view video, take notes, and complete quizzes to 


demonstrate knowledge and understanding. 
Students can use the interactive notes 
feature to type notes. Notes are displayed 


Students can play and 
next to the corresponding video frame. 


review video content. 


] ‘Cams and rivers are common locations generating 
is Wnydrostectricity. 


& Energy trom the wood releases only carbon, 


| Uhanol and tuodiesel are ewo forms of biofuel 
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"thane watched the video. 
Ham ready te eit and take quis. 


felt the instrection 


SOURCE: Delen, E. (2013). Scaffolding and enhancing learners’ self-regulated learning: Testing the effects of online video-based interactive 
learning environment on learning outcomes. (Doctoral dissertation). https://doi.org/1969.1/151123 
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Self-monitoring prompts embedded in online 
activities boost students’ comprehension by 
activating prior knowledge and encouraging 
them to organize their thinking while learning. 
These prompts also support students when 
applying their knowledge in different activities 
or learning environments. Structured note 
taking can facilitate information collection 

and academic achievement in online learning 
environments, and prompting students to 
self-monitor themselves during note taking 
helps them filter information and deepen 
understanding as they move along ina lesson.*? 


Content creation and delivery tools can support 
self-regulated learning. Such tools include those 
for instructors to upload a course syllabus, course 
content, and assignments, and for students 

to access course resources and readings. They 
do so by giving students what they need to 
engage in the processes of goal setting, help 
seeking, self-evaluation, and selecting strategies 
for completing tasks. Communication and 
collaboration tools can help foster interaction 
and help seeking by establishing an open and 
friendly community of learners. Administrative 
tools such as calendars and web-based to-do 
lists can support students’ time planning and 
management skills.* 


There are several strategies for sending students 
reminders and prompts—sometimes called 
“nudge” messages—that provide tips and coach 
students to use self-regulated learning strategies. 
Email and text-based reminders are one way 

to encourage students to stay on track while 


completing course assignments and studying 
for exams. Examples of text-based reminders and 
prompts are provided in Figure 3.2. 


Nudge messages are being built into adaptive 
learning environments; smartphone apps and 
student support software are being designed to 
provide students coaching and reminders to use 
self-regulated learning strategies throughout the 
semester. For example, some apps and software 
provide students with access to information about 
assignments, campus resources and supports, and 
availability of course materials. These tools can 
also be used to track students’ progress toward 
goals, such as on-time attendance. 


Assessment tools, such as those to post grades 
and track student progress, can support students’ 
task strategies, self-monitoring, and self- 
evaluation.*? Some technologies can facilitate 
automated monitoring of student progress and 
can generate alerts or messages to instructors 
and/or students. 


Messages to students can be personalized to 
offer encouragement and support during the 
semester to both students who are thriving 
and those who might be at risk for failure. For 
example, messages generated by E*?Coach 
(Figure 3.3), a software-based student-support 
system, advise students by sending personalized 
messages and information about how their 
performance compares to their classmates’ The 
personalized messages include information 
about goals and values the individual students 
identified at the beginning of the semester. 
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FIGURE 3.2. SAMPLE TEXT-BASED REMINDERS AND PROMPTS TO SUPPORT 
SELF-REGULATED LEARNING 


Text-based reminders and metacognitive prompts are designed to provide tips and to coach students to 
use self-regulated learning strategies as they complete course assignments and study for exams. 


Subject: Review your 
forum posts 


é ; Look back on your forum 
Subject: Both Checklists posts so far. Ask - Have | 


Remember, Review included all required 

checklists for lessons 2 components? How do my 

and 3. slen time every responses compare to 

day. You'll need it! others? What can | do better 
next time? 


Subject: Two Quizzes! 


Quizzes 2 and 3 are due Subject: Plan for 

this week. Keep a level the coming week 
workload. Complete Ask yourself - What are the 
quiz 2 the first half of the topics for this week? How 
class week & quiz 3 by the = much time will | need each 
end of the class week. day? Preview checklists. Is 


there a forum this week? 
Subject: Brainstorm 


speech topics Subject: Final exam 

You must prepare & present Begin preparing for the final 
three speeches (special exam now. As you study for 
occasion, informative, & weekly quizzes, keep notes 
persuasive). Open a new and materials organized for 
document, list each type, quick review later. It is 10% 


think, & type. Just do it! of your grade! 


SOURCE: Adapted from Sullivan, S. M. (2016). The effects of prompting metacognition using email or text reminders on student participation, 
persistence, and performance in a blended course (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses. (Accession 
No. 10102309) 
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FIGURE 3.3. SAMPLE INDIVIDUALIZED EMAIL BASED ON STUDENT GOALS 


AND PERFORMANCE 


E*Coach can send individualized messages to students based on their course goals and performance. This 
message is tailored to a student whose first course exam is approaching. 


Hello Alan, First! Name 


E*Coach is here to give you pointers about how to approach studying for this first exam. Remember that this advice 
comes from over 49,000 previous students and the entire physics teaching faculty. These are techniques that people 
just like you recommend over and over, and they'll help you work towards a strong first exam score so you can build 
towards receiving a Be like you want. Grade_Want 


Preparing for a midterm can be kind of overwhelming and even intimidating, especially when it’s your first 


E&M exam in college! You might be asking yourself: Course 


Will the 9 hours | was planning on spending be enough? Hours_Exam_Study 


Should | work with a friend? 


How do | know what material to focus on? 


What if | don’t do as well as | want to? 


Before you check out the other tabs for study tips, take a minute to remind yourself of what 


you value. You said that you value relationships with family and friends, independence and learning 


and gaining knowledge. Then you wrote about why: 


Values_Aff_Select 


BY ie 
x 


=S 
= 


| believe it is our experiences that make up who we are, and through which we gain value, 

and purpose. We make these experiences with our relationships, and our independence. | 

value my ability to make decisions for myself, and to share these decisions with others. However, 
with the knowledge necessary to make decisions, | will make mistakes. | hope that by having 


a good basis of knowledge and understanding Ican pursue experiences with my family and 


friends that | can look back on. __ values_Aff_Response 
—————— 


Keep this in mind as you begin to prepare for your first exam. 


You can be successful on this exam if you commit to putting in the hard work, time, and effort. 


SOURCE: https://er.educause.edu/articles/2013/12/e2coach-tailoring-support-for-students-in-introductory-stem-courses 


2. Model how to use self-regulated learning 
strategies, and provide students with 
opportunities to practice self-regulated 
learning strategies using technology. 


Students may need instruction on self-regulated 
learning strategies, including what they are, 

why they are important, and how to use them. 
Campus resource centers should offer students 
training, videos, and other resources on self- 
regulated learning strategies. Strategies might 
include time management tips, study skills, 
concept mapping tools, and note-taking and 
comprehension aids, such as those in Figure 3.4. 


Instructors and advisors should also become well 
versed in these resources so they can encourage 
students to access, learn, and apply self-regulated 
learning strategies. Instructors should model the 


use of self-regulated learning strategies. They 
should provide explicit guidance and examples 
of strategies and behaviors used by students who 
successfully navigated the course in previous 
semesters, thus providing “best practices” to 
adopt. Recommendation 5 (“Teach students how 
to become self-regulated learners”) in the Practice 
Guide Strategies for Postsecondary Students in 
Developmental Education* offers tips for teaching 
students in developmental education courses 
how to enact self-regulated learning strategies. 


Students need opportunities to practice self- 
regulated learning strategies before they can 
get into the habit of applying them. Allowing 
students to build on prior knowledge through 
practice supports the transfer and application of 
knowledge and skills across settings or learning 
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FIGURE 3.4. EXAMPLE OF CAMPUS RESOURCES TO SUPPORT SELF-REGULATED LEARNING 


LSU's Virtual Learning 
Center offers interactive 
resources to help students 
achieve their academic 
goals. Workshops feature 
success strategies and 
interactive quizzes. 


STUDENT ENTRY & 
BB GUEST ENTRY » 
Campuses can offer 
students resources for time ®-—® 
management, planning, 


preparing for class, and 
identifying strengths. 


© Wideo: How | Leam 
+ Video Learning by Styfe 
Video Auditory Lemners 


Virtual Learning Center 


Discover Strengths 


Learning Joumey 


Marve sos 


Time Management 


+ Map My Success 
* Quiz: Am | using time wisely? 

* Video: Procrastination Prevention 
© Video: Tools for Managing 
© Video: Where Does My Tim 
© Quiz: Dol organize & plan 


and Cookies are enabled to mack your progress 


Planning 


Guat your week on Sunday 


* Weeldy Calender 
Sunday Stort 


* Weeldy Calender 


e comtoit € 


Reduce Stress 


* Reduce My Stress 
* Video: Deep Breathing 


» Video Visualization 
© Video: What is Strees? 


study cycle Go! 


Monday Start 


Learning support sites (like LSU’s) 
enable students to find specific 
resources related to studying. 


THE STUDY CYCLE 


The Path to Improving Study Techniques 


FOCUSED STUDY SESSIONS 


Focused Study Sessions (FSSs) are designed to work with the way your 
brain learns best: in short, focused increments. 
Schedule several focused study sessions per class each week. 


Decide what you will accomplish in your 
study session and get started. 


(Suggested time: 1 - 2 minutes) 


Interact with material: organize, concept 
map, summarize, process, read, work 
problems. 


(Suggested time: 30 - 50 minutes) 


Step away from material to clear 
your head. 


(Suggested time: 5 - 10 minutes) 


Go back over, summarize, wrap-up and 
check what you studied. 


(Suggested time: 5 minutes) 


* Should | continue studying? 
* Should | take a break? 


Adapted from Frank Christ's PLRS system. 
©2015 Louisiana State University, Center for Academic Success 


CHOOSE 


* Should I change tasks or subject? 


Division of 
I Su | Student Affairs 
Center for Academic Success 


B-31 Coates Hall * (225)578-2872 « Isu.edu/cas 


SOURCE: http://Isu.edu/students/cas/makebettergrades/successresources/virtuallearningcenter.php 


USING TECHNOLOGY TO SUPPORT POSTSECONDARY STUDENT LEARNING | RECOMMENDATION 3: INCORPORATE TECHNOLOGY 32 
THAT MODELS AND FOSTERS SELF-REGULATED LEARNING STRATEGIES 


RECOMMENDATION 3 


contexts. Instructors should encourage students 
to draw on prior knowledge as they are learning 
new information or solving unfamiliar problems. 


support college students’ self-regulation through 
a range of learning technologies. The instructor 
can use this training model to guide students 


through four phases of a learning task—from 


Kitsantas and colleagues (2015) offer a four- observation to self-regulation. 


level model of self-regulation development (see 
Figure 3.5) that can be used as a training tool to 


FIGURE 3.5. SAMPLE TRAINING MODEL TO SUPPORT SELF-REGULATION THROUGH 
LEARNING TECHNOLOGIES 


Instructor identifies the learning task; determines the processes, steps, or procedure associated with 
performing the learning task; and then uses the four-phase training model. 


Instructor uses synchronous collaborative tools (e.g., whiteboards, chat 
session) or asynchronous tools (e.g., a webcast, podcast, video tools) to 
demonstrate or model the step-by-step process or procedure associated 
with performing the learning task. 


PHASE 1: OBSERVATION 


Enables students to think 
strategically before acting 


Student observes the instructor and begins to discriminate between 
strategies used to execute the task. 


Student uses synchronous collaborative tools (e.g., whiteboard, chat) 
or asynchronous tools (e.g., a webcast, podcast, video tools) to emulate 
or enact the task in the presence of the instructor and peers to receive 


information about their feedback and coaching. 


learning behaviors and 
performance with the help 


of the instructor and peers 4 


Instructor coaches and provides encouragement and feedback, 
gradually fading this support based on student performance. 


Student uses wikis, blogs, document-sharing technologies, and online 
bookmarking tools to practice performing the learning task on his or her 
own and monitors and rewards own progress. 


PHASE 3: SELF-CONTROL 


Enables students to focus 
on mastering the steps 
related to completing the 
learning task during practice 
episodes 


Instructor provides scaffolding for student self-monitoring through the 
commenting features of wikis and blogs and the collaborative editing 
and tagging features of document-sharing and online bookmarking 
technologies. 


PHASE 2: EMULATION 
Enables students to gather 


Student refines the task using already selected learning technologies 
and creates a personal learning environment to document the 
self-regulatory strategies used to achieve the task outcomes. 


PHASE 4: SELF-REGULATION 
Enables students to focus 
on outcomes and make 
strategic adjustments 

when performing poorly Instructor is available for feedback, if needed; instructor uses 
synchronous and asynchronous collaborative technologies to 


provide feedback. 


SOURCE: Adapted from Kitsantas et al. (2015), p. 286 
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3. Select technologies that feature tutoring or 


mentoring components to support students 
in using self-regulated learning strategies. 


Traditional face-to-face tutoring can be especially 
helpful for students who are in online or blended 
learning settings.*’ Tutors can serve as a scaffold 
by providing supports and helping students 
enact various aspects of self-regulated learning. 
Aspects might include planning their learning, 
monitoring their understanding, and applying 
different strategies to learn content. Emerging 
technologies, such as intelligent tutoring 

systems, cognitive tutors, and adaptive learning 
environments, can offer promising alternatives 

to face-to-face adaptive tutoring, which can be 
resource intensive. 


Pedagogical agents can be embedded in 
computer-based learning environments to 
complement and extend other scaffolds such as 
searchable text, simulations, and concept maps. 
MetaTutor, for example, is an adaptive learning 
environment that includes pedagogical agents 
to scaffold students’ use of self-regulated learning 
processes while learning challenging science 
topics. The interface elements in MetaTutor (see 
Figure 3.6) support the following: 


e Identification of an overall learning goal for the 
learning session, which is set by the instructor; 


e Selection of subgoals, which are set by 
the student; 


e Anorganized table of contents to help 
students navigate the content; 


e Selection of self-regulated learning processes 
to use during the learning session; and 


e Pedagogical agents to support the student 
with selected self-regulated learning strategies 
during the learning session: 


» As the students make their way through 
the content, which is displayed through 
text and static images, the pedagogical 
agent prompts them to use the selected 
self-regulated learning processes. 


» Prompts, including suggestions and 
questions, are recorded in the Dialog 
History box. Students may review the 
Human-Agent dialog history during the 
session and save it for future reference. 


» The Input Area is where students can take 
notes and submit answers to questions. 


» Periodically, the pedagogical agent uses the 
Input Area to prompt students to reflect on 
the content they are learning and submit 
a metacognitive judgment of their level of 
understanding of that content. 


4. Assess whether the selected technologies are 


effectively supporting students in their use of 
self-regulated learning strategies. 


Most online environments, including course and 
learning management systems, track students’ 
time spent in various activities or functions. To 
the extent possible, instructors should monitor 
which functions and features students are using 
most frequently. Instructors should poll students 
to determine whether the technologies, such as 
note-taking features, self-monitoring prompts, 
and feedback features, are helping them apply 
specific self-regulated learning strategies. 
Instructors can also examine usage data and 
student performance to see if there is any 
association between the two. 


nstructors can also use online usage data to 
help students understand how classmates 
interacting with similar features to the same 
degree/frequency are performing in the course. 
For example, the Check My Activity tool at 
University of Maryland, Baltimore County was 
designed to be a “self-service feedback tool” for 
students. Students can compare their online 
course activity against a class average and can 
also check their activity and frequency of use 
against an anonymous summary of peers who 
earned the same, higher, or lower grade for any 
assignment (see Figure 3.7). 
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FIGURE 3.6. EXAMPLE OF ADAPTIVE LEARNING ENVIRONMENT WITH PEDAGOGICAL AGENTS 
TO SUPPORT SELF-REGULATED LEARNING 


OVERALL LEARNING GOAL 
A learning goal (set by the 
experimenter or teacher). 


TIMER 
Shows time remaining 
in learning session. 


SUBGOALS 
The learner can generate several 
subgoals for the sessions. 


MetaTutor 
{Learning Goal and Subgoals Zz _—<_oococ™c™c™ CY AGENT 
Resists fo lean all you can about SiciRtory Sytemepccticly Be siesa| lea pbout all os are Th f 
organs ai r Components of the circulatory system, and their purpose he syffem, how they wor 
Midualy and together and how they support the heathy funct Se are ouh 
TABLE OF Q==—@ Table of Contents 5 ; pedagogical 
mE Lungs nn 3] | earthea (Complete Subgoal ‘ 
CONTENTS page eenteeses ae] agents assigned 
oneaene Hae bn |: Malfunctions of the circulatory system (Prioritize Subgoal to different 
Blood Role — 
subtopics of Parisereioeda| | (Heart Beat = wm lieearning SRL processes. 
. Subgoal 
the content. BloodPlasma | Heart: Heart Beat (1/2) eS 
| 
pocd Bed lce | |Unlike most muscles that are directed by nerve Prior Knowledge SRL PALETTE 
Blood: Hemoglo! impulses, the heart muscle (myocardium) can Judgement of Learning L 
. '|contract by itself. Specialized heart muscle cells - - earners ma 
Bleou whrteBie also have two important functions: (1) to make Feeling of Knowing y 
Blood: Platelets electrical signals that spread to the rest of the Content Evaluation enacta ny of these 
Blood Vessels | |heart and cause it to contract with a regular, Monitoring Pi 
lesen renee beat, and (2) to provide a conduction ahd: fesse oe SRL processes 
| Coordination of i i 
Vessels: Veins = pee during learning. 
- Vessels: Capillar Drawing and Note 
/Heart | ': Okay, you are now ready to summarize. Please type your sum#ary in the box below. Inferences 
lember, you should try to include as much information from thacontent as you understood. = 
Cycle of Blood FI Do you want to add this summary to your notes for this pag DIALOG 
: Ok, it looks like you want to summarize. Go ahead and type itn to the box below. And Summarization 
_ Cycle of Blood FI mber to include as much info as you understood. Pinal pirecteal Senich HISTORY 
Heart Valves : Great! You can add that summary to your notes if you want nfjw. 
: Please type your summarization of this content in the box beJpw. Remember to try to. — Hu man-Agent 
Heart Beat ||| givalas much information as you understood from this content as pifssible. é F 
Heart BeatCont : Now that yo have completed a sumary, would you like to adajthis summary to your dia log h istory. 
is? 
Diastole/Systole : Please rate the level of your understanding. 
Diastole/Systole 
__ + Systems of Circulation Ido not understand the information in this page @ lunderstand fe information in this page i 
. Other Aspects of LCS I vaguely understand the information jghis page O [know the infflrmation in this page Submit 
cl > ms | somewhat understand the informatiol} in this page © Lam familiar will the information in this page 
TEXT CONTENT AREA IMAGE CONTENT AREA 
Science content. Static representations 


of content. 


LEARNER’S 
METACOGNITIVE 
JUDGEMENTS AREA 


SOURCE: Azevedo, R., Moos, D.C., Johnson, A.M., & Chauncey, A.D. (2010). Measuring Cognitive and Metacognitive Regulatory 


Processes During Hypermedia Learning: Issues and Challenges. Educational Psychologist, 45 (4), 210 — 223. 
http://dx.doi.org/10.1080/00461520.2010.515934 
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FIGURE 3.7. SAMPLE TOOL FOR STUDENTS TO MONITOR PROGRESS: CHECK MY ACTIVITY 


e The University of Maryland, Baltimore County (UMBC) uses a Blackboard learning management system (LMS) to 
support faculty and students. To supplement the LMS, the university created a custom Check My Activity (CMA) 
tool that allows learners to compare their own level of activity in Blackboard Learn (number of sessions, number 
of clicks by tool type, amount of time in class) against an anonymous summary of course peers. If instructors use 
the LMS grade book, students can also compare their own activity with peers earning the same, higher, or lower 
grade on any assignment. 


Start Topics ~~ © Events. © Community Groups +P 


e By providing students with the 
ability to benchmark their activity 
against others in their same classes, 
underperforming students are 
encouraged to spend more time 
engaging with course materials and 
developing effective study habits. 
Using Check My Activity can help 
raise awareness that fosters 
self-regulated learning. 


« Back to Chock My Activity & 


ctivity for LLC 646: 


= ¥ 48 in this course. 
‘= This course's activity ranks 2nd out of § LLC courses in Blackboard. 


Course Usage (Average Hits per User) 
Toot Hite 
‘ 300 
71.00 


SS 
a 


] Below fea Average | | Above 


e Students can see how active they are on Blackboard compared to others, as this feature 
highlights their number of visits to the site compared to the class's total visits. 

e The color green represents falling within the average, while yellow represents 20% 

below and blue represents 20% above. 

e This feature also shows what the course average is and how the individual student 
compares. 


Information on site Your Usage 


use is also made 


easily acccessible Tool 
in bar charts. Discussion Board 
“Other” may ce Other 
messaging, calendars - 

ging, d Send Email 
announcements, 
and other interactive Tools 
features. Total 


SOURCE: http://blog.blackboard.com/student-facing-learning-analytics-self-regulated-learning 
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Potential Obstacles and the OBSTACLE: Instructors might be uncertain about 
Panel’s Advice the best way to incorporate self-regulated learning 


strategies into their specific courses. 


OBSTACLE: /nstructors do not believe self-regulated 


oe PANEL'S ADVICE: Incorporating self-regulated 
learning is something they should have to teach 


learning strategies into classroom activities does 


college students. not need to be burdensome or time-consuming. 
PANEL'S ADVICE: Many instructors believe McGuire (2015) offers several approaches 

students should already know how to monitor for instructors and suggests students can be 

their learning, and many students do not believe introduced to self-regulated learning strategies in 
they need to be taught how to self-regulate their an hour-long workshop. Instructional designers and 
learning. Administrators should incorporate self- professional development staff can assist instructors 
regulated learning into their institution-wide in articulating the goals and objectives for their 
learning objectives. They should share evidence that course or for the activities within their course. 


self-regulated learning is associated with positive 
student outcomes and that technology is an 
effective way to support self-regulated learning. To 
further underscore the institution’s commitment to 


nstructional designers can also help instructors 
identify strategies and actions students can take to 
meet these goals and objectives. 


encouraging self-regulated learning, administrators Next designers can help instructors think about 
can also encourage instructors to integrate how to provide opportunities for students to 
discipline-specific versions of the objective into earn—and practice—self-regulated learning 
their courses. strategies throughout the semester. As a follow- 


up to workshops, instructional designers should 
consider one-on-one meetings with instructors 
to familiarize them with technologies available to 
scaffold students’ self-regulated learning and to 
teach them how to integrate these technologies 
into their courses. 


Professional development programs for instructors 
should include training on the importance of 
self-regulated learning skills. They are especially 
important in computer-based learning 
environments and activities, which often involve less 
guidance and support from the instructor. Training 


can provide instructors with examples of how to OBSTACLE: Web-based technologies designed to 
incorporate and model the use of self-regulated support selfregulated learning can lack interfaces that 
learning strategies in their courses. are intuitive and efficient for learners to navigate. 
Students can vary in their degree of comfort and PANEL'S ADVICE: When designing or selecting 
readiness for working in a self-regulated learning web-based activities or modules, instructors and 
environment. They should be offered university- instructional designers should ensure each interface 
and department-level training. The training is designed to allow students to effectively apply 
should provide students explicit instruction in self-regulated learning strategies. To ensure students 
self-regulated learning strategies and help them are maximizing their time and attention focusing 
understand how to effectively apply the strategies on the relevant content, it is essential to make 
in their learning. Once instructors and students sure learners are not over-burdened by learning 
understand the importance of self-regulated how to use new tools and interfaces. Students and 
learning skills and how students can apply them, instructors should have access to instructions, an 
they might be more receptive to exploring and orientation, and a help feature to ensure they are 
engaging with the scaffolding features embedded aware of and able to efficiently use all functions 
within many of the technologies they already use. within the technology. 

USING TECHNOLOGY TO SUPPORT POSTSECONDARY STUDENT LEARNING | RECOMMENDATION 3: INCORPORATE TECHNOLOGY 37 


THAT MODELS AND FOSTERS SELF-REGULATED LEARNING STRATEGIES 


RECOMMENDATION 4 


Recommendation 4: Use technology to provide timely and 
targeted feedback on student performance. 


The panel recommends using technology to 
facilitate the provision of timely and targeted 
feedback to students about their performance. 
Specifically, technology is recommended as a tool 
to assess students’ skills and understanding and 
to deliver the results as feedback to students and 
instructors. Students learn more effectively when 
they are actively engaged in learning activities 
and have direct access to information about what 
they are doing right or wrong.** However, some 
college classes provide limited opportunities during 
class for students to actively engage with content 
and demonstrate their skills or understanding of 
key concepts. This is especially common in large 
or lecture-based classes. Deep understanding 

and retention of material can be difficult when 
opportunities to apply new knowledge are not 
provided concurrent with its delivery. Outside of 
class, students can more effectively prepare for 
exams and for the next class if they have access to 
information about how well they have mastered the 
content and its application.*° 


Technology permits rapid assessment and 
tabulation of student responses and can be used 
both in and outside of class. Thus, technology 
provides opportunities for students to engage with 
content and demonstrate their understanding, or 
lack of understanding, to themselves and to their 
instructor at the time that learning activities occur. 
Technology that facilitates assessment and the 
provision of timely feedback, whether in or outside 
of class, with individual students or with groups, also 
provides data that can be used to tailor and modify 
instruction to better address students’ learning 
needs and challenges. 


Among the most widely used technologies in 
traditional face-to-face courses for providing 
immediate feedback are automated student 
response systems. These can be hand-held devices 
or polling applications for smartphones or tablets/ 
laptops that allow students to respond to questions; 
software then rapidly tabulates responses and 
shows the results to the class so students can 
check their answers and see how they compare 

to those of their peers. Outside of class time, or in 
blended and online courses, timely feedback can 
be provided via online assessments, assignments, 
or polling applications. Feedback systems can also 
be incorporated into online courses and modules 
by using embedded assessments; many learning 
management systems have these options built 

in. Developments in artificial intelligence enable 
feedback to be individually tailored via adaptive 
systems, such as intelligent tutoring systems. 


With all of these technologies, the rapid tabulation 
of data permits students to check their performance 
quickly and allows instruction to be adapted 

and differentiated in light of student responses. 

This differentiation can be performed by human 
instructors or can be programmed into online 
courses and modules. 


WWC staff and the expert panel assigned a 
moderate level of evidence based on eight studies. 
Five studies meet WWC group design standards 
without reservations.*° Two studies meet WWC 
group design standards with reservations.*' One 
study that tested multiple interventions reported 
on comparisons that meet group design standards 
without reservations and comparisons that meet 
group design standards with reservations.” See 
Appendix C for a detailed rationale for the Level of 
Evidence for Recommendation 4. 
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SPOTLIGHT ON IMPLEMENTATION: USING ONLINE HOMEWORK AND STUDENT 
RESPONSE SYSTEMS TO PROVIDE TIMELY AND TARGETED FEEDBACK TO STUDENTS 


An instructor teaching a developmental pre-algebra course at a community college used online assignments 
and student response systems to provide timely and targeted feedback to students about their performance. 
The class met twice a week. 


Online homework assignments provided the instructor with student information to plan the next class 
session. Prior to each class, students were assigned online homework and were expected to complete it in 
advance. Although students did not receive scores on the homework assignments prior to class, the online 
homework assignment platform allowed students to practice concepts and get help with the homework 
problems. Each class section started with a review, where the instructor was able to address students’ 
questions on their previous homework assignment. 


The instructor followed a three-segment format during class to assess student understanding of previously 
taught and new content in real time: 


e First, the instructor used a student response system to review the material covered in the previous session, 
using rapid-fire questions from the homework. The instructor typically showed 12-15 problems on the 
screen. Students worked in pairs to solve the problems and recorded their answers with their individual 
devices. After a reasonable amount of time, the instructor posted the distribution of responses on the 
screen and then demonstrated the solution. Particularly difficult concepts were often highlighted by 
using more than one question. 


e The second segment of class focused on introducing new concepts. After each new concept was 
introduced, students were given additional problems to solve using their devices. These questions were 
designed using an easy-hard or easy-hard-hard format. The first question was designed so that most 
students would get the answer correct and was intended to build their confidence. The second and third 
questions were more difficult; the second question was often similarly constructed to the instructor's 
examples, while the third question required students to apply the concept in a novel or unfamiliar way. As 
with the homework review, students could work together and answers were posted on the screen. The 
instructor then illustrated the solution step-by-step. 


e Finally, students could begin work on the new assignment while the instructor was available for help. The 
student response system platform used in the course was a radio frequency system called Turning Point® 
that was purchased by the college. This platform is compatible with Microsoft PowerPoint®. The online 
homework was provided by the textbook publisher through a system called Course Compass°®. 


Study Findings: Moreau (2009) reports the use of the student response system had indeterminate effects on 
student achievement. 
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How to Carry Out 
the Recommendation 


The guidance below is informed by the studies 
that support the recommendation, as well as the 
expert panel's expertise and knowledge of 
resources and strategies available to help 
implement the recommendation. 


1. Determine the course segments or content 
for which students would benefit most from 
timely and targeted feedback. 


In planning a course syllabus, instructors 

should identify in advance common mistakes 
and “bottleneck concepts,’ the foundational 
concepts that, when students find them difficult 
to understand, prevent them from grasping 
more complex, related concepts. Students 

and instructors benefit from timely feedback 
about performance related to concepts that are 
commonly misunderstood by students.*? Exams 
and homework from previous terms can be 
mined to identify such concepts. Class discussions 
and student questions can also help identify the 
nature of common student misunderstandings. 


Armed with this information, instructors can 
then select the concepts to cover and plan the 
timing of assessment, such as polling in class 

or giving quizzes prior to class meetings. They 
also can plan the type and format of feedback 
(group or individual) best suited to helping 
students understand their mistakes or get past 
the bottleneck concepts. For example, in-class 
polling can be more appropriate for remediating 
common misconceptions that can be addressed 
quickly by the instructor in real time—for 
example, the difference between correlation and 
causality in a statistics course. On more complex 


bottleneck concepts, instructors might be able to 
better tailor class meetings, and students might 
benefit more from those classes, if assessment 
and feedback on those concepts occur prior to 
the class meeting. After determining the course 
segments and content for which students would 
benefit from timely and targeted feedback, 
instructors can decide whether technology could 
be useful to facilitate the collection of student 
responses and provision of feedback. 


Decide which technologies to use for 
providing feedback. 


After deciding to use technology to check 
students’ understanding and provide timely, 
targeted feedback, instructors must select the 
tools that best align with their course format, 
the content about which feedback will be 
provided, and the institutional context in which 
they operate. When selecting technologies to 
provide feedback to students, instructors should 
determine the availability of technology, as well 
as which technologies other instructors and 
students currently use. 


Face-to-face learning environments can 
incorporate technology-based questioning and 
real-time tabulation of data from the entire class. 
For polling or quizzing students, older-style 
student response systems, often called ‘clickers’ 
might be available for classroom use. There are, 
however, newer web- and smartphone-based 
applications that can perform the same tasks, 
and students might feel more comfortable 
using their laptops or smartphones to respond 
to questions. Poll Everywhere (see Figure 4.1) 
allows instructors to create questions before 
class, which they then administer and can 
immediately see the results of during class. 


USING TECHNOLOGY TO SUPPORT POSTSECONDARY STUDENT LEARNING | RECOMMENDATION 4: USE TECHNOLOGY TO 


PROVIDE TIMELY AND TARGETED FEEDBACK ON STUDENT PERFORMANCE 


RECOMMENDATION 4 


FIGURE 4.1. SAMPLE CLASSROOM POLLING TOOL: POLL EVERYWHERE 


Create questions before 
your presentation 


SOURCE: https://www.polleverywhere.com/how-it-works 


Not all technologies designed to provide 
students with in-class feedback require individual 
student devices. Instructors are the only ones 
who need a smartphone or tablet to use 

Plickers (paper clickers), which allow them to 
conduct real-time checks for understanding and 
to administer “exit tickets” (impromptu polls). 


FIGURE 4.2. SAMPLE CLASSROOM 
RESPONSE SYSTEM: PLICKERS 


Oso" 


| #7) Daily grammar 1 (co. 


69% correct total: 1 3/21 


Talsie-lahanyaee)|(-velt 
student responses 


SOURCE: https://itunes.apple.com/us/app/ 
plickers/id701184049?mt=8 


TaWAltom eke] aa cel] ey=]a1 6) 
to respond 


View the results 
lamactcl ma laats) 


Students receive a paper card, which they can 
hold up to be scanned by the instructor using 
an app on a smartphone or tablet. Data are 
automatically saved for individual students. 


In online courses, especially those that are 
delivered asynchronously, it might be more 
feasible to provide immediate feedback to 
students through automated scoring of 

online homework or quizzes. Online learning 
environments can be designed to provide 
students immediate feedback on navigational 
choices and on answers they provide to 
questions posed throughout a lesson, in online 
homework, or in quizzes. 


Many of the adaptive learning environments 

have built-in assessments and can provide 

timely feedback. The Open Learning Initiative, 
introduced in Recommendation 2, offers free, 
open access to course materials that provide 
immediate feedback. The feedback helps students 
assess their own learning and study effectively 
while also providing instructors with immediate 
grading and feedback on students’ progress. 


Strategically incorporate feedback 
technologies into the course. 


Instructors must make decisions about how 

to incorporate feedback into their lectures 

and course units. The physical organization, 
features, and amenities of the classroom, such as 
seating arrangements and availability of wireless 
internet, for example, might influence choices 
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about which technologies can be used for 
in-class assessment and feedback. At the same 
time, class size and class duration should be 
considered when planning how to incorporate 
technology-based feedback into a face-to-face 
course, as these factors can influence how much 
time the instructor dedicates to discussion of 
student responses. 


Students can respond to questions individually 
or in small groups, and instructors can choose 
to provide opportunities for students to receive 
both individual and group feedback. When 
used effectively, technology-based questioning 
techniques not only provide opportunities for 
giving timely feedback, but also support active 
learning and engagement during class. 


Instructors teaching online or blended courses 
should work with instructional designers to build 
in knowledge checks that call on students to 
apply and demonstrate their understanding of 
course material. Online homework or quizzes 
with automated grading features offer students 
and instructors immediate feedback on student 
performance that takes place outside of class. 


In all types of course formats, technologies 
should be selected to provide feedback that 
aligns with course objectives and goals. Students 
should understand whether and how feedback 
plays into their assignment and course grades. 
Instructors using student response systems 

for in-class activities must decide whether or 
not to assign credit to feedback generated 
from technology-driven checks for student 
understanding. In deciding, they should 

weigh the benefits of allowing students to 
anonymously demonstrate their understanding 
versus a desire to use the results for attendance 
and/or grading. Ungraded feedback allows 
students to practice demonstrating their 
knowledge without fear of consequence if they 
answer incorrectly, but making participation 
count toward the student's grade can promote 
engagement with the assessment. Instructors 
of online courses should work with instructional 
designers to decide how frequently feedback 
should be provided in content modules. 


4. Design questions that align with the desired 


learning objectives. 


Instructors must make decisions about what 
content and technology are best suited 

for assessing students and providing them 

with feedback. They also must construct the 
questions they will ask in class with student 
response systems, include on online homework 
assignments, or use in online assessments. Before 
drafting questions, instructors should identify 
learning objectives for the activity or unit that 
will include feedback. Instructors can draw on 
their understanding of common misconceptions 
about the material—informed by students’ past 
work, questions, and performance—to create 
questions and answer choices that are plausible 
enough to challenge students to work through 
those misconceptions.* 


The optimal difficulty level of questions can vary 
for different groups of students, and instructors 
should pilot and refine questions. Rather than 
posing single questions to students, instructors 
should use easy-hard-hard question sequences,” 
where the easy question builds confidence, the 
first hard question introduces cognitive conflict, 
and the second hard question introduces a 
different context. 


Instructional designers and teaching and 
learning center staff can help instructors 

write questions at the higher levels of Bloom's 
Taxonomy in order to extend their assessment 
beyond knowledge-level multiple-choice 
questions. Instructors should especially consider 
interpretive and analytic (why, how, and in your 
opinion) questions rather than recall (who, what, 
or when) questions. Figure 4.3 provides an 
overview of different questions instructors can 
ask using student response systems. 


When designing multiple-choice questions, 
instructors should include incorrect response 
options that elicit common misunderstandings, 
which create ready-made opportunities to 
correct them. Tactics for meeting a variety of 
question design goals that instructors may 
have when using student response systems are 
presented in Figure 4.4. 
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FIGURE 4.3. TYPES OF QUESTIONS THAT CAN BE USED WITH CLASSROOM RESPONSE SYSTEMS 


8 OD @ 


CLASSROOM EXPERIMENTS RECALL QUESTIONS CONCEPTUAL UNDERSTANDING QUESTIONS 
conducted by faculty or other ask students to recall facts, concepts, go beyond recall and assess students’ understanding 
researchers can be facilitated by or techniques and don't require of important concepts. They are often based on 
collecting data from higher-order thinking skills. common student misconceptions. 
students using classroom 
response systems. 


Mi, 


MONITORING QUESTIONS APPLICATION QUESTIONS 
are designed to provide instructors require students to apply their 


with information about how their 
students are approaching the 
learning process in their courses. 
For example, asking students how 
long they took to complete an 
assignment they have just turned 
in can provide instructors with 
useful information about the difficulty 
of the assignment. 


knowledge and understanding 
to particular situations and contexts. 
These questions often ask students to 
make a decision or choice in a given 
scenario and can connect course 
content to “real-world” situations. 


6 6 © 


CONFIDENCE LEVEL QUESTIONS STUDENT PERSPECTIVE QUESTIONS CRITICAL THINKING QUESTIONS 
are asking following content questions. ask students to share their opinions, operate at the higher levels of Bloom's Taxonomy, 
Asking students to rate their confidence in experiences, or demographic information. These requiring students to analyze relationships among 
their answers (high, medium, or low) questions do not have correct answers and can multiple concepts or make evaluations based on 
can aid in metacognition—learning often generate rich discussion, particularly particular criteria. These questions can be very 
about one’s own learning. questions about ethical, legal, or moral issues. The effective in preparing students to engage in 
anonymity that clickers provide is often an essential class discussions about their reasons. 
ingredient in asking these kinds of questions. 


SOURCE: https://cft.vanderbilt.edu/cft/guides-sub-pages/clickers/ 
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FIGURE 4.4. EXAMPLE DESIGN TACTICS FOR CREATING CLICKER QUESTIONS 


QUESTION DESIGN GOALS TACTICS 


e Remove potentially distracting features and steps (i.e., nonessentials) 


e Use scenarios and features that lead students to compare and contrast 


e Extend the context; ask familiar questions of unfamiliar scenarios 


DIRECT ATTENTION Reuse familiar question situations; new ideas should be introduced in 


AND RAISE AWARENESS familiar question scenarios 

e Use questions to draw students into making an error then follow up with a 
question that leads them to realize their mistake, and allow them to go back 
“Oops-go-back") 


e Interpret representations; provide necessary information or answer options 
in alternative representations 


e Identify a set or subset of situations, objects, or processes 


e Rank variants of situations, objects, or processes according to some 
STIMULATE COGNITIVE measurable quality 


seis e lee e Constrain the solution by instructing students to use, or avoid, certain 


problem solving approaches 
e Reveal better problem solving approaches 


e Present a problem and ask students to identify the strategy that would most 
effectively reach a solution 


e Push students to identify what information is necessary to solve a problem 
by including extraneous information and omitting necessary information 


e Use answer choices that reveal likely difficulties including common errors, 
misunderstandings, and alternative assumptions and interpretations 


e Use“none of the above” answer choices, and make it the correct answer 
often enough that it is useful 


FORMATIVE USE OF 
RESPONSE DATA 
e Use qualitative questions that promote discussion of concepts and ideas 
e Use analysis and reasoning questions that require substantial decision making 
e Use questions with multiple defensible answers that require unstated 
assumptions, trap unjustified assumptions, and/or are deliberately 
aeMOlING ambiguous to generate discussion 
ARTICULATION/ 
DISCUSSION e Design questions that deliberately catch students in common 


misconceptions 


SOURCE: Beatty et al. (2006) 
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Yourstone and colleagues (2008) suggest that 
instructors should allow students in face-to- 
face courses to discuss individual answers to 
questions after they have submitted them 

but before the correct answer is revealed. 

Doing so promotes peer learning and deeper 
understanding. Students also benefit from 

time to discuss and reflect on both correct 

and incorrect responses. Instructors can take 
advantage of “teachable moments” by reviewing 
all answer choices, especially those that suggest 
students are not grasping the content. 


Depending on the platform used by individual 
online learning modules, feedback can be 
corrective or explanatory or both. Corrective 
feedback provides the correct answer when a 
student selects an incorrect answer; explanatory 
feedback goes a step further by providing an 


explanation of the correct and incorrect answers. 


Both corrective and explanatory feedback can 
be effective for supporting student learning, 
depending on the material.“ 


Use data to inform instruction, and to help 
students guide their learning. 


Instructors collecting real-time data about 
students’ understanding during a face-to-face 
class or in a synchronous online session can 
implement audience-paced instruction. Such 
instruction means the instructor can decide 
whether to continue a lecture/discussion or 
pause to elaborate on concepts students are not 
yet mastering. 


Data collected from student response systems 
can be tabulated quickly for the instructor, 

and the instructor can decide whether or not 
to graphically display a summary of the data. 
Instructors can use webpages or chat functions 
that allow students to anonymously post 
questions and comments during a class. Then 
the instructor can periodically pause to check 
for questions and comments and address any 
without individual students feeling on the spot 
for raising the question or comment. 


Data gathered through online assessments 
conducted during lessons, especially if tabulated 
quickly for the instructor, can highlight concepts 
the instructor should review with students 
before moving to the next lesson or unit in 

the course. If shared with students, these data 
can also help students understand how their 
performance compares to their peers: 


When providing timely and targeted feedback to 
students, instructors also can provide guidance 
on how students can use data to improve their 
future performance. For example, instructors 
might suggest students revisit course materials 
or they might recommend new content to 
study. Instructors can leverage data and analytic 
capabilities built into many technologies to 
improve their use of the technology over time. 
Data collected during class can be saved and 
analyzed to assess question efficacy, as well as to 
modify questions for future semesters. 


Work collaboratively to adopt and integrate 
newer technologies. 


Many technologies are available to provide 
students timely, targeted feedback during 
in-class sessions. Even more sophisticated 
technologies are being developed to provide 
students with feedback that supports 
metacognitive thinking and reflection when 
they are online, such as adaptive learning 
environments, interactive online modules 

and courses, and intelligent tutoring systems. 
Instructors, instructional designers, and software 
developers can collaborate to develop and adopt 
these technologies. Open Learning Initiative is 
one such collaborative opportunity. It invites 
instructors to help develop or evaluate courses 
that the Initiative is currently building. 


Education technology startup companies also 
can be good partners for instructional designers 
and instructors who are interested in integrating 
new technologies into their classrooms.” The 
federally supported Small Business Innovation 
Research program also provides opportunities 
to receive grants for conducting research and 
development on innovative technologies. 
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Potential Obstacles and the 
Panel’s Advice* 


OBSTACLE: Creating questions that support timely 
and targeted feedback can be time-consuming. 


PANEL'S ADVICE: Many resources are available to 
instructors interested in incorporating timely and 
targeted feedback into their courses. Textbooks 
often come with supplementary materials, 
including assessment questions; some websites 
offer freely available question banks or libraries. 
Many course and learning management systems 
provide software that allows instructors to develop 
and maintain question banks or question pools. 
Instructors can team up with other instructors 
teaching similar courses to co-develop or exchange 
questions. Students should be invited to submit 
questions for in-class discussion and review. 
Questions submitted in advance of the class can 
be reviewed by the instructor, and they can also be 
saved for use with future groups of students. 


OBSTACLE: Lecture time is limited, and adding 
real-time checks for understanding and follow-up 
discussion can be time-consuming. 


PANEL'S ADVICE: Instructors should be judicious in 
their use of real-time polling and feedback. Instead 
of feeling obligated to dedicate large amounts of 
time to this type of assessment, instructors can 


leverage feedback technologies selectively. For 
example, they can use these technologies when 
covering particularly complex topics, to break 

up a lengthy lecture, or as a way to check for 
understanding at the beginning or end of a class. 


OBSTACLE: Adapting instruction during a lesson 
based on student responses can be difficult, especially 
for instructors with less teaching experience or 
familiarity with the content they are teaching. 


PANEL'S ADVICE: For even the most experienced 
instructors, preparation is key. Instructors should 
be fully prepared to discuss both the correct and 
incorrect answer choices for all questions posed to 
students. They should also be prepared to explain 
or reteach content when students demonstrate 
misunderstandings through in-class polling or on 
quick assessments. Peer-led discussion may also 
be an effective strategy for helping students work 
through misunderstandings.” 


When instructors do not feel sufficiently prepared 

to immediately adjust instruction based on 

student responses, they can review data and 

make adjustments before the next class session. 
Administrators can play a key role here by 
connecting instructors with the necessary resources, 
such as professional development and teaching and 
learning centers, to support this preparation. 
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Recommendation 5: Use simulation technologies that help 
students engage in complex problem-solving. 


The panel recommends the use of technologies through challenging situations where they are 
such as computer simulations, game-based safe to be wrong without consequences. These 
learning, and virtual reality environments that technologies can allow students to take multiple 
simulate and allow students to grapple with approaches to solving a problem and help them 
complex problems. Incorporating complex understand the implications of the approach they 
problem-solving activities into postsecondary decide to take by simulating consequences. 
instruction allows students to interact more 

This recommendation emphasizes the use of 


deeply with learning material, to practice 
higher-order thinking skills, and to make 
connections among concepts. These activities 
can promote engagement, which enhances 
student understanding and improves retention of 
material.°° Problem-solving activities such as case 
studies or field projects are common in higher 
education. Often, however, the amount of time 
students have to solve a problem within a typical 
semester is limited, or they might not be able to 
test and explore the results of multiple solutions.” 
As technology has evolved, new options that 
provide realistic and immersive problem situations, 
support testing and analysis of multiple scenarios, 
and involve social interactions among students 
have emerged. These are promising strategies for 
engaging students in complex problem-solving. 


technology that is designed to feature realistic, 
complex problems that help students deepen their 
understanding and develop higher-order thinking 
skills. Technologies that simulate complex problems 
might include one or more of the following features: 
allow learners to make decisions and observe the 
outcomes of their decisions, provide an opportunity 
for students to practice newly acquired skills, 
facilitate problem-based learning, and allow for 
immersive role playing. 


WWC staff and the panel assigned a minimal level 
of evidence based on two studies that examined 
the effects of a web-based simulated problem- 
based learning experience. One study meets WWC 
group design standards without reservations” 
and one study meets WWC group design 
standards with reservations.°? See Appendix C for 
a detailed rationale for the Level of Evidence for 
Recommendation 5. 


Technologies that simulate complex problems 
can mitigate the time and place constraints of 
traditional case studies or field projects. They also 
provide students with the opportunity to work 
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SPOTLIGHT ON IMPLEMENTATION: COMMUNITY PUBLIC HEALTH SIMULATION 


Two instructors of an undergraduate Health Behavior course implemented and tested a semester-long 
internet-based simulation of a community public health problem with a group of undergraduate public health 
majors (Spinello & Fischbach, 2004). The simulation was designed to give students the opportunity to use 
problem-solving and critical thinking to develop their own solution to a poorly defined, authentic problem. 


For their lesson, the professors used a web-based platform to simulate the effects of a mock disease outbreak 
in a virtual community. The simulation had students act as public health officials and explore the problem 
through newspaper clippings and television media, through the use of simulated epidemiological data, and 
with surveys or interviews with characters in the simulation, such as medical professionals, reporters, patients. 
The online, media-rich environment gave the simulation a realistic look and feel and allowed the students to 
explore the problem from multiple perspectives and test intervention strategies. The simulation mimicked real- 
world problems by unfolding over time, so that students were required to work on solutions without having 
the full picture. The instructor could also change the environment by inserting features into the simulation, 
such as a request for an interview from a local reporter or phone messages from local physicians or family 
members. In addition, there was a simulated conference room that permitted the students to communicate 
asynchronously and allowed the instructor to pose as a simulation character in such meetings. 


Students worked in small groups to address the virtual outbreak, and each action or intervention taken by the 
student groups resulted in instructor-implemented changes in the environment. In this way, students could 
see the results of their interventions and propose modifications or adjustments to their plans. During the 
course, for example, one student group elected not to respond to a request from a journalist for an interview. 
In response, the local news station announced plans to run an expose on the health department's bungled 
attempts to address the problem. 


Study Findings: The impact of the web-based community health simulation on academic achievement was 
positive and statistically significant. 


the time to engage with authentic, complex 
problems or the focus instead is on mastery of 
introductory core concepts. In planning and 
implementing successful simulated learning 
experiences, instructor and student preparation 
are important, as is including time for post- 
experience discussion and reflection. 


How to Carry Out 
the Recommendation 


The guidance below is informed by the studies that 
support the recommendation, as well as the expert 
panel's expertise and knowledge of resources 

and strategies available to help implement the 


recommendation. 2. Establish the learning objectives first, and 


1. Assess and decide whether a course then select an activity with the right level 


offers the appropriate context for using 
technologies that simulate complex 
problems to promote higher-order thinking 
and problem-solving skills. 


Not all courses are suited for including complex 
simulations or problem-based learning 
activities. Instructors and instructional designers 
should consider whether the curriculum 

and content of the course can give students 


of complexity. 


Instructors and instructional designers should 
first determine the goals they hope to achieve 
through technologies that simulate complex 
problems, and then select an activity or case that 
can help to accomplish those goals. Simulated 
problems should be aligned to both the learning 
content and the desired learning outcomes of 
the course. 
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Instructors should consider whether they want to achieve the learning objective or one that 
students to work independently or in small increases in complexity as learners achieve deeper 
groups, and then identify technologies that have understanding of the material. See Figure 5.1 for 
been successfully used by other instructors in an example of an online simulation tool available 
that setting. They should select a technology to instructors and instructional designers. 


that has the appropriate amount of complexity 


FIGURE 5.1. SAMPLE ONLINE SIMULATION TOOL: CALIFORNIA BUDGET CHALLENGE 


BUDGET OVERVIEW 
Check out this section of 
the website to learn more. 


. a Heath & Human Sennces 


CalWORKs & Child Care 
Spending Current Policy 


The budget refiects a reduction © funding tot CahWORKs and chid 
W127 Caunsmen care programs 


CALIFO@NIA BLIDGET 


Commas Colleges 


Cut $0.4 billion 
Raextuoe fueuting for each sot and lower'ng the ncome at ehich 
nabainy tamées are chigtic . THE METER 
uce osu . Keeps track of the 
Cut $0.7 billion mans OCOMe budget you are 


Hone Cary Reduce gars archer % anc init grants tor Pulten 48 monts building for California 


Cut $1.1 billion 


Further reducing funding tor bath) CalWORKs and chit care 


Spend $1.8 billion 
Reskre te redcufions ade to CatvORKs and child care in he 
March Ducijet 


SUMMARY 

See a summary of the 
budget you are building. 
You can also print or 
About email your budget to the 
Health & Human Services CalWORKs & Childcare Governor, family, friends 


and colleagues. 
Cer Neat) and tuenan serve programs are expected to cost 


sbout $14 1 Délon m 2010-11 about $5 4 of the budget 


TELL US TAKE ACTION ABOUT PROS & CONS 
If you still do not see You can use the Take Every Policy option Each policy choice contains 
a policy choice you Action button at any time contains an About section a section of Pros & Cons 
are looking for or have to share your choices just below your choices. where you can learn both 
an option you think and comments with the sides of an argument. 
we should research Governor and your state 
and consider adding, representatives via email. 
please tell us. 


SOURCE: https://www.budgetchallenge.org/challenges/12/pages/instructions 
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3. Introduce students to expectations for the 
simulated learning activity, and provide 
supports to ensure they feel comfortable in 
the learning environment. 


Instructors should be clear and transparent 
about the outcomes students are expected to 
achieve through the simulated learning activity. 
Instructors also should explain the features of 
the exercise that will help students achieve these 
outcomes. When introducing the objectives 

and expectations for the task, instructors should 
clearly explain how students will be assessed and 
graded. If students will be working in groups or 


teams, expectations about the contributions of 
individuals within the group should be clear. 


Instructors also should explain that team-based 
learning focused on solving a complex problem 
is an opportunity for students to develop 
collaborative decision-making skills. If possible, 
each team member should be provided with a 
specific role or set of responsibilities for engaging 
with the activity. Strategies used to structure 
learning activities that require teams of students 
to engage with complex problems are featured 
in Figure 5.2. 


FIGURE 5.2. SAMPLE TIPS FOR GROUP PROBLEM-BASED LEARNING ASSIGNMENTS USING 


IMMERSIVE TECHNOLOGY 


1. Convey the expectation that students work collaboratively to solve or manage the problem. 


2. If students are collaborating online, encourage the use of synchronous collaboration tools such as group 
chat, shared whiteboards, video conferencing, and group browsing. 


3. Make the learning objectives of the assignment readily accessible to the students. 


4. Give students the ability to negotiate their own learning needs in the context of the assignment. 


5. If possible, assign tutors to work directly with the teams to provide support and facilitate discussion (but 
not to direct discussion). This tutor can also plan real-time collaboration sessions using the synchronous 


tools mentioned previously. 


SOURCE: Savin-Baden, M., Tombs, C., Poulton, T., Conradi, E., Kavia, S., Burden, D., & Beaumont, C. (2011). An evaluation of 
implementing problem-based learning scenarios in an immersive virtual world. International Journal of Medical Education, 2, 116. 


Before allowing students to begin such learning 
activities, instructors should make sure the 
technology is working and there are no other 
barriers to navigating it. This includes explaining 
how to use the technology and where students 
can get help should technology-related 
questions arise. 


4. Lead students in reflective discussion to help 
them evaluate their own learning. 


Instructors should help students to understand 
there might not be a “right” answer to simulated 
problems presented in activities, especially 
because students bring diverse perspectives to 
situations. Students should be encouraged to 
be comfortable with ambiguity and failure and 


should be guided to reflect on and learn from 
their mistakes. Instructors should encourage 
collaboration so that students take advantage of 
the opportunity to learn from the experiences of 
peers engaging with the same complex problem. 


Students participating in simulated learning 
activities are not directly provided solutions 

to the problems at hand. Learning occurs as 

they construct and apply their own solutions. 
Instructors can facilitate that learning by 

guiding students to complete short reflections or 
assessments throughout the process. For example: 


e Rubrics and questionnaires can guide 
students to reflect on learning-related 
activities, challenges, and accomplishments. 
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e Think-aloud protocols can help instructors 
understand what learners are thinking as 
they engage with both the content and 
the technology. 


Other feedback mechanisms, such as questions 
about students’ confidence level or level 

of interest, can be embedded throughout 
simulated learning activities. Data about 
students’ experiences with an activity can be 
used to inform instructors’ decisions about 
whether and how to use the simulated activity 
again with a different group of students. 


5. Keep an eye out for new or evolving 
technologies that support effective, 
engaging, complex problem-solving. 


Emerging immersive, virtual world, 

augmented- and mixed-reality, and artificial 
intelligence technologies all offer exciting 
learning opportunities for students to work 
together on complex problems that simulate 
real-world scenarios. Evidence on the impacts 

of these emerging technologies on learning is 
scant thus far. Educators should keep an eye 

out for information on how these cutting-edge 
technologies can be applied to foster learning in 
face-to-face, online, or blended courses. Teaching 
and learning centers on campus can be excellent 
resources for identifying new technologies. 


Potential Obstacles and the 
Panel’s Advice 


OBSTACLE: Many instructors follow lesson plans 

they have previously developed, and some might be 
reluctant to give up time they normally spend lecturing 
to make room for simulated problem-solving activities. 


PANEL'S ADVICE: Instructors can blend complex 
problem-solving simulations with course content 

in a way that does not require a wholesale redesign 
of their course. They can incorporate these activities 
into their courses gradually without taking up too 
much lecture time, by assigning students to prepare 


for the activity before class and then allowing time 
for students to debrief about the simulated problem 
in the context of a lesson being taught during class. 


Depending on the instructor's goals and the 
complexity of the simulated activity, instructional 
designers can estimate how much time it would 
take to develop a new simulation or customize an 
existing one. After implementing the simulated 
learning activity the first time, the instructor can 
weigh the outcomes, benefits, and tradeoffs of 
using technology to simulate complex problems in 
the course, and then consider whether or not to use 
the same or different technology in the future. 


OBSTACLE: /t can be challenging and time- 
consuming to design and implement effective lessons 
using technology that simulates complex problems. 


PANEL'S ADVICE: There are many off-the-shelf 
simulations that instructors can use. Instructional 
designers can provide professional development to 
instructors and help them think through objectives, 
resources, and time constraints to select the most 
appropriate learning technology to simulate complex 
problems. Teaching centers and instructional 
designers can also provide instructors with strategies 
to integrate simulated activities into other course 
activities and assignments. There are also emerging 
tools and authoring environments that can help 
instructors more easily co-develop and implement 
simulated problems with instructional designers. 


OBSTACLE: The open-ended nature of simulated 
learning environments can make it particularly 
challenging for students to engage with and 
solve problems. 


PANEL'S ADVICE: Instructors and instructional 
designers should select simulated activities that 
offer clear guidance, instructions, and readily 
accessible help features for both students and 
instructors. When possible, instructors should use 
their understanding of student needs and learning 
preferences to inform their selection of simulated 
learning activities. 
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THAT 


HELP STUDENTS ENGAGE IN COMPLEX PROBLEM-SOLVING 


RECOMMENDATION 5 


Instructors can ask their students about their past in groups, instructors can pair more experienced 
experiences with technology-based simulated students with less experienced students. Practice 
activities and use that information to tailor sessions can also be helpful for students who are 
instructions and guidance to fit the experience struggling with the technology. 


level of the class. If students are expected to work 
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A 


ADAPTIVE LEARNING is an educational method that uses computer algorithms to coordinate the interaction 
with the learner and deliver customized resources and learning activities to address the unique needs of each 
learner. Computers adapt the presentation of educational material according to students’ learning needs, as 
indicated by their responses to questions, tasks, and experiences. 


ADMINISTRATOR refers to staff at postsecondary institutions in leadership positions, such as deans, provosts, 
and department heads. These individuals might assist with the implementation of recommendations in this 
guide by providing resources and supports to instructors, instructional designers, advisors, and other staff who 
use technology to support postsecondary student learning. 


ARTIFICIAL INTELLIGENCE (Al) refers to computer systems that undertake tasks usually thought to require 
human cognitive processes and decision-making capabilities. To exhibit intelligence, computers apply algorithms 
to find patterns in large amounts of data—a process called machine learning, which plays a key role in a number 
of Al applications. Al learning agents have the potential to function like adaptive learning but at a much more 
sophisticated and nuanced level. 


AUGMENTED REALITY is an enhanced version of reality created by the use of technology to overlay digital 
information on an image of something being viewed through a device (such as a smartphone camera). This can 
also refer to the technology used to create augmented reality. 


BLENDED LEARNING courses combine face-to-face classroom instruction with online learning, reducing 
classroom contact hours. Blended learning is sometimes referred to as hybrid learning. 


BLOOM'S TAXONOMY is a classification system used to define and distinguish different levels of human 
cognition—that is, thinking, learning, and understanding. Educators have typically used Bloom's taxonomy to 
inform or guide the development of assessments (tests and other evaluations of student learning), curriculum 
(units, lessons, projects, and other learning activities), and instructional methods such as questioning strategies. 
The original taxonomy (developed in the 1950s) was updated in 2001 (Anderson & Krathwohl, 2001), when 
three categories were renamed and all the categories were expressed as verbs rather than nouns. The revised 
categories are: Remembering, Understanding, Applying, Analyzing, Evaluating, and Creating. 


C 


COGNITIVE LOAD refers to the amount of information that the human brain's working memory can hold at one 
time. Because working memory has a limited capacity, instructional methods should avoid overloading it with 
additional activities that don't directly contribute to learning. 


COMPUTER SIMULATIONS involve the usage of a computer for the imitation of a real-world process or system. 


CONCEPT MAPS are diagrams or graphical tools that visually represent relationships among concepts and ideas. 
Most concept maps depict ideas as boxes or circles (also called nodes), which are structured hierarchically and 
connected with lines or arrows (also called arcs). These lines are labeled with linking words and phrases to help 
explain the nature of the connections. A concept mapping tool is used to create concept maps. 
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COURSE OR LEARNING MANAGEMENT SYSTEM refers to a web-based system for the administration, 
documentation, tracking, reporting, and delivery of educational courses or training programs. Such systems are 
used by instructors to manage and deliver course materials to students, administer tests and other assignments, 
track student progress, and manage record keeping. Instructors can also use these systems to communicate with 
students, to share course materials, and to encourage discussion among students. Popular examples of these 
systems include Blackboard, Canvas, and Moodle. 


E 


ENGAGEMENT refers to the degree of attention, curiosity, interest, optimism, and passion that students show 
when they are learning or being taught, which extends to the level of motivation they have to learn and progress 
in their education. 


G 


GAME-BASED LEARNING is a type of game play that has defined learning outcomes. Generally, game-based 
learning is designed to balance subject matter with game play and the ability of the player to retain and 
apply said subject matter to the real world. Three main elements of game-based learning are competition, 
engagement, and immediate rewards. 


IMMERSIVE TECHNOLOGY refers to technology that blurs the line between the physical world and digital or 
simulated world, thereby creating a sense of immersion. Immersive technology enables mixed reality. 


IMMERSIVE VIRTUAL WORLD refers to a simulated environment, facilitated by networked computers, that 
provides multiple users with avatars and communication tools with which to act and interact in real-time. 


INSTRUCTIONAL DESIGNERS assist with designing and redesigning courses and academic programs. Their 
major task is to work with faculty and other instructional designers to create pedagogically appropriate material 
such as content, media, and manuals that are used in teaching and learning. They are cognizant of and rely on 
learning theory to develop material that will appeal to and engage students in their studies. 


INSTRUCTOR refers to any individual who teaches full- or part-time at the postsecondary level. This includes 
faculty and adjunct professors; it can also include graduate students and other staff responsible for teaching 
courses. 


INTELLIGENT TUTORING SYSTEM refers to a computer system that aims to provide immediate and customized 
instruction or feedback to learners, usually without requiring intervention from a human teacher. Such systems 
have the common goal of enabling learning in a meaningful and effective manner by using a variety of 
computing technologies. An intelligent tutoring system typically aims to replicate the demonstrated benefits 

of one-to-one, personalized tutoring in contexts where students would otherwise have access to one-to-many 
instruction from a single teacher (e.g., classroom lectures) or no teacher at all (e.g., online homework). 


M 


META-ANALYSIS is the statistical procedure for systematically combining data from two or more similar studies 
to develop a conclusion that has greater statistical power than either study alone. 
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MIXED REALITY is the merging of real and virtual worlds to produce new environments and visualizations 
where physical and digital objects coexist and interact in real time. Mixed reality uses immersive technology. 


MODULES organize course content by weeks, units, or some other organizational structure. Each module can 
contain files, discussions, assignments, quizzes, and other learning materials. A course is made up of one or more 
modules packaged together. 


O 


OPEN EDUCATIONAL RESOURCES (OER) are freely accessible, openly licensed text, media, and other digital 
assets that are useful for teaching, learning, and assessing, as well as for research purposes. OER describes 
publicly accessible materials and resources for any user to use, re-mix, improve, and redistribute under 

some licenses. 


P 


PEDAGOGICAL AGENT is a concept borrowed from computer science and artificial intelligence and applied 
to education, usually as part of an intelligent tutoring system. It is a simulated human-like interface between 
the learner and the content, in an educational environment. A pedagogical agent is designed to model the 
interactions between a student and another person. A pedagogical agent can be assigned various roles in the 
learning environment, such as tutor or co-learner, depending on the desired purpose of the agent. 


S 


SCAFFOLDING is an instructional process through which supports for students are added in order to enhance 
their learning and aid in their mastery of tasks. This is done by systematically building on students’ experiences 
and knowledge as they are learning new skills. As students master the assigned tasks, the supports are gradually 
removed so they gain greater independence in the learning process. 


SELF-REGULATED LEARNING is a cyclical process, wherein the student plans for a task, monitors his/her 
performance, and then reflects on the outcome. The cycle then repeats as the student uses the reflection 
to adjust and prepare for the next task. The process is not one-size-fits-all; it should be tailored for individual 
students and for specific learning tasks. 


SOCIAL LEARNING THEORY is the belief that people learn from one another, via observation, imitation, 
and modeling. 


V 


VIRTUAL REALITY is an artificial environment experienced through sensory stimuli (such as sights and sounds) 
provided by a computer and in which one’s actions partially determine what happens in the environment. This 
can also refer to the technology used to create or access a virtual reality. 


VIRTUAL WORLD refers to a computer-based online community environment designed and shared by 
individuals so they can interact in a custom-built, simulated world. Users interact with one another in this 
simulated world using text-based, two-dimensional, or three-dimensional graphical models called avatars. Avatars 
are graphically rendered using computer graphics imaging or any other rendering technology. Individuals control 
their avatar using their keyboard, mouse, and other specially designed command and simulation gadgets. Today's 
virtual worlds are purpose-built for entertainment, social, educational, and training purposes. 
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Appendix A: Postscript From the Institute of 
Education Sciences 


What Is a Practice Guide? 


The Institute of Education Sciences (IES) publishes practice guides to share evidence and expert guidance on 
addressing education-related challenges not readily solved with a single program, policy, or practice. Each 
practice guide's panel of experts develops recommendations for a coherent approach to a multifaceted problem. 
Each recommendation is explicitly connected to supporting evidence. Using What Works Clearinghouse™ 
(WWC) group design standards and WWC pilot regression discontinuity standards, the supporting evidence is 
rated to reflect how well the research demonstrates the effectiveness of the recommended practices. 


Strong evidence means positive findings are demonstrated in multiple well-designed, well-executed studies, 
leaving little or no doubt that the positive effects are caused by the recommended practice. Moderate evidence 
means well-designed studies show positive impacts, but there are questions about whether the findings can 
be generalized beyond the study samples or whether the studies definitively show evidence that the practice 
is effective. Minimal evidence means that there is not definitive evidence that the recommended practice is 
effective in improving the outcome of interest, although there might be data to suggest a correlation between 
the practice and the outcome of interest. (See Table A.1 for more details on levels of evidence.) 


How Are Practice Guides Developed? 


To produce a practice guide, IES first selects a topic. Topic selection is informed by inquiries and requests to the 
WWC Help Desk, a limited literature search, and evaluation of the topic’s evidence base. Next, IES recruits a panel 
chair who has a national reputation and expertise in the topic. The chair, working with IES and WWC staff, then 
selects panelists to co-author the guide. Panelists are selected based on their expertise in the topic area and the 
belief that they can work together to develop relevant, evidence-based recommendations. Panels include two 
practitioners with expertise in the topic. 


Relevant studies are identified through panel recommendations and a systematic literature search. These studies 
are then reviewed against the WWC group design standards by certified reviewers who rate each effectiveness 
study. The panel synthesizes the evidence into recommendations. WWC staff summarize the research and help 
draft the practice guide. 


IES practice guides are then subjected to external peer review. This review is done independently of the IES 
staff that supported the development of the guide. A critical task of the peer reviewers of a practice guide is 
to determine whether the evidence cited in support of particular recommendations is up-to-date and that 
studies of similar or better quality that point in a different direction have not been overlooked. Peer reviewers 
also evaluate whether the level of evidence category assigned to each recommendation is appropriate. After 
the review, a practice guide is revised to meet any concerns of the reviewers and to gain the approval of the 
standards and review staff at IES. 


Institute of Education Sciences Levels of Evidence for What Works 
Clearinghouse Practice Guides 


This section provides information about the role of evidence in IES's WWC practice guides. It describes how 
practice guide panels determine the level of evidence for each recommendation and explains the criteria for 
each of the three levels of evidence (strong evidence, moderate evidence, and minimal evidence). 
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The level of evidence assigned to each recommendation in this practice guide represents the panel's judgment 
of the quality of the existing research to support a claim that, when these practices were implemented in past 
research, positive effects were observed on student outcomes. After careful review of the studies supporting 
each recommendation, panelists determine the level of evidence for each recommendation using the criteria 
in Table A.1. The panel first considers the relevance of individual studies to the recommendation and then 
discusses the entire evidence base, taking the following into consideration: 

e The number of studies 

e The study designs 

e The internal validity of the studies 

e Whether the studies represent the range of participants and settings on which the recommendation is focused 


e Whether findings from the studies can be attributed to the recommended practice 


e Whether findings in the studies are consistently positive 


A rating of strong evidence refers to consistent evidence that the recommended strategies, programs, or practices 
improve student outcomes for a diverse population of students.* In other words, there is strong causal and 
generalizable evidence. 


A rating of moderate evidence refers either to evidence from studies that allow strong causal conclusions but 
cannot be generalized with assurance to the population on which a recommendation is focused (perhaps 
because the findings have not been widely replicated) or to evidence from studies that are generalizable 
but have some causal ambiguity. It also might be that the studies that exist do not specifically examine the 
outcomes of interest in the practice guide, although the studies might be related to the recommendation. 


A rating of minimal evidence suggests that the panel cannot point to a body of evidence that demonstrates the 
practice's positive effect on student achievement. In some cases, this simply means that the recommended 
practices would be difficult to study in a rigorous, experimental fashion’; in other cases, it means that researchers 
have not yet studied this practice, or that there is weak or conflicting evidence of effectiveness. A minimal 
evidence rating does not indicate that the recommendation is any less important than other recommendations 
with a strong or moderate evidence rating. 


In developing the levels of evidence, the panel considers each of the criteria in Table A.1. The level of evidence 
rating is determined by the lowest rating achieved for any individual criterion. Thus, for a recommendation to get 
a strong rating, the research must be rated as strong on each criterion. If at least one criterion receives a rating of 
moderate and none receives a rating of minimal, then the level of evidence is determined to be moderate. If one 
or more criteria receive a rating of minimal, then the level of evidence is determined to be minimal. 


The panel relied on WWC group design standards and WWC pilot regression discontinuity standards to assess 
the quality of evidence supporting education programs and practices. The WWC evaluates evidence for the 
causal validity of instructional programs and practices according to WWC group design standards. Information 
about these design standards is available at https://whatworks.ed.gov. Eligible studies that meet WWC group 
design standards without reservations or meet WWC group design standards with reservations are indicated by 
bold text in the endnotes and references pages. 


* Following WWC guidelines, improved outcomes are indicated by either a positive, statistically significant effect or a positive, 
substantively important effect size. The WWC defines substantively important, or large, effects on outcomes to be those with effect sizes 
greater than or equal to 0.25 standard deviations. See the WWC guidelines at https://whatworks.ed.gov. 


T For more information, see the WWC Frequently Asked Questions page for practice guides at https://whatworks.ed.gov. 
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A Final Note About IES Practice Guides 


In policy and other arenas, expert panels typically try to build a consensus, forging statements that all its 
members endorse. Practice guides do more than find common ground; they create a list of actionable 
recommendations. Where research clearly shows which practices are effective, the panelists use this evidence 
to guide their recommendations. However, in some cases, research does not provide a clear indication of what 
works. In these cases, the panelists’ interpretation of the existing (but incomplete) evidence plays an important 
role in guiding the recommendations. As a result, it is possible that two teams of recognized experts working 
independently to produce a practice guide on the same topic would come to different conclusions. 


Institute of Education Sciences 
Table A.1. IES levels of evidence for What Works Clearinghouse practice guides 


EVIDENCE BASE 


CRITERIA 
Validity 


STRONG 


High internal validity (high- 
quality causal designs). Studies 
must meet WWC design 
standards with or without 
reservations. 


AND 


High external validity (requires 
multiple studies with high- 
quality causal designs that 
represent the population on 
which the recommendation is 
focused). Studies must meet 
WWC design standards with or 
without reservations. 


MODERATE 


High internal validity but 
moderate external validity 
(Le, studies that support 
strong causal conclusions, but 
generalization is uncertain). 


OR 


High external validity bu 
moderate internal validity 

(Le, studies that support the 
generality of a relation, but the 
causality is uncertain).° 


MINIMAL 


The research may include 
evidence from studies that 
do not meet the criteria for 
moderate or strong evidence 
(for example, case studies, 
qualitative research). 


Effects on 
relevant 
outcomes 


Consistent positive effects 
without contradictory evidence 
(.e., No statistically significant 
negative effects) in studies with 
high internal validity. 


A preponderance of evidence 
of positive effects. Contradictory 
evidence (i.e, statistically 
significant negative effects) 
must be discussed by the panel 
and considered with regard 

to relevance to the scope of 

the guide and intensity of 

the recommendation as a 
component of the intervention 
evaluated. 


There may be weak or 
contradictory evidence 
of effects. 


Relevance 
to scope 


Direct relevance to scope 

(i.e, ecological validity)— 
relevant context (for example, 
classroom vs. laboratory), 
sample (for example, age 

and characteristics), and 
outcomes evaluated. 


Relevance to scope (ecological 
validity) may vary, including 
relevant context (for example, 
classroom vs. laboratory), 
sample (for example, age and 
characteristics), and outcomes 
evaluated. At least some research 
is directly relevant to scope (but 
the research that is relevant to 
scope does not qualify as strong 
with respect to validity). 


The research may be out of the 
scope of the practice guide. 
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CRITERIA 


Relationship 
between 

research and the 
recommendations 


STRONG 


Direct test of the 
recommendation in the studies 
or the recommendation is 

a major component of the 
intervention tested in the 
studies. 


EVIDENCE BASE 
MODERATE 


Intensity of the recommendation 
as a component of the 
interventions evaluated in the 
studies may vary. 


MINIMAL 


Studies for which the intensity 
of the recommendation as a 
component of the interventions 
evaluated in the studies is low; 
and/or the recommendation 
reflects expert opinion based on 
reasonable extrapolations from 
research. 


Panel confidence 


The panel has a high degree of 
confidence that this practice is 
effective. 


The panel determines that 

the research does not rise to 
the level of strong but is more 
compelling than a minimal level 
of evidence. 


The panel may not be confident 
about whether the research has 
effectively controlled for other 
explanations or whether the 
practice would be effective in 
most or all contexts. 


In the panel's opinion, the 
recommendation must be 
addressed as part of the 
practice guide; however, the 
panel cannot point to a body of 
research that rises to the level of 
moderate or strong. 


Role of expert 
opinion 


Not applicable. 


Not applicable. 


Expert opinion based on 
defensible interpretations of 
theory (theories). (In some 
cases, this simply means that 
the recommended practices 
would be difficult to study in a 
igorous, experimental fashion; 
in other cases, it means that 
esearchers have not yet studied 
this practice.) 


When assessment 
is the focus of the 
recommendation 


For assessments, meets the 
standards of The Standards for 
Educational and Psychological 
Testing. 


For assessments, evidence 

of reliability that meets The 
Standards for Educational 

and Psychological Testing 

but with evidence of validity 
from samples not adequately 
representative of the population 
on which the recommendation 
is focused. 


ot applicable. 


This includes randomized controlled trials (RCTs) and quasi-experimental design studies (QEDs). Studies not contributing to levels of evidence include 
single-case designs (SCDs) evaluated with WWC pilot SCD standards and regression discontinuity designs (RDDs) evaluated with pilot RDD standards. 


>The research may include studies generally meeting WWC group design standards and supporting the effectiveness of a program, practice, or approach 
with small sample sizes and/or other conditions of implementation or analysis that limit generalizability. The research may include studies that support the 
generality of a relation but do not meet WWC group design standards due only to lack of demonstrated equivalence at pretest for QEDs. QEDs without 
equivalence must include a pretest covariate as a statistical control for selection bias. These studies must be accompanied by at least one relevant study 


meeting WWC design standards. For this practice guide, the latter studies did not need to be considered because a sufficient number of studies meet WWC 


design standards for each recommendation. 


© American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999). 
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Appendix B: Methods and Processes for Developing This 
Practice Guide 


Phase 1: Selecting the Topic and the Panel; Establishing a Review Protocol 


The initial step in the development of a practice guide, as described in Appendix A, is the selection of the topic. 
WWC staff* nominated practice guide topics based on their relevance to postsecondary education practitioners 
and the amount of rigorous research available. To inform the level of rigorous research, WWC staff conducted a 
preliminary literature search and screened abstracts to identify topics where a critical mass of rigorous evidence 
might exist. The relevance to postsecondary educators was informed by consultations with postsecondary 
education advisors. After the topic of “using technology to support postsecondary student learning” was nominated 
for this guide and selected by IES, a team of nationally recognized experts was selected to form the panel. 


EXPERT PANEL. This practice guide was developed by WWC staff in conjunction with the expert panel. The panel 
comprised Dr. Nada Dabbagh, George Mason University; Dr. Randall Bass, Georgetown University; Dr. MJ Bishop, 
University System of Maryland; Dr. Anthony G. Picciano, City University of New York; and Dr. Jennifer Sparrow, Penn 
State University. The recommendations in the practice guide drew on the panel's nationally recognized expertise 
and its balance of researcher and practitioner experience in the field of instructional technology. 


SCOPE AND PURPOSE. The panel and WWC staff initially defined the scope and purpose of the practice guide 
by focusing on the following question: What are the most promising technologies associated with improving 
postsecondary student learning outcomes? Using this guiding question, WWC staff worked with the panel to 
identify the types of technologies that would be covered by the practice guide; describe how those technologies 
support learning and instruction; and identify the parameters to be placed on the research included in the 
practice guide. WWC staff then codified these decisions into a practice guide review protocol. 


PRACTICE GUIDE REVIEW PROTOCOL. The review protocol defines the following: the purpose of the practice 
guide; interventions, populations, and outcomes covered by eligible research; the evidence criteria to be defined 
in reviews; and procedures for conducting the literature search. The protocol specifies that studies used as 
evidence will examine interventions that use technology to support student learning in various ways. In addition, 
to be eligible for the review, a research study had to: 


e use acomparison group design (e.g., a randomized control trial or a quasi-experimental design); 
e involve college students in the United States; 
e be published in 1997 or later; and 


e report on one or more outcomes in the following domains: (1) academic achievement, (2) college 
attendance, (3) credit accumulation and persistence, (4) attainment of a degree, certificate, or credential, (5) 
post-college employment and income, or (6) student engagement and motivation. 


Phase 2: Literature Search and Review 


A comprehensive literature search was conducted to identify all studies potentially relevant to the practice guide. 
The search focused on studies published between 1997 and 2017 that examined practices for using technology 
to support learning in postsecondary settings. The studies were primarily identified through keyword searches of 
several databases, and supplemented with additional studies recommended by the expert panel. 
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The literature search generated more than 50,000 studies. All studies were screened for eligibility for the practice 
guide, using the criteria defined in the review protocol. Eligible studies were then reviewed by WWC-certified staff 
against the standards defined in the What Works Clearinghouse Procedures and Standards Handbook, Version 3.0.°° 
Once the review was complete, studies were assigned one of the following evidence ratings: 

e Meets WWC Group Design Standards Without Reservations; 

e Meets WWC Group Design Standards With Reservations; or 

e Does Not Meet WWC Group Design Standards. 


Studies that met WWC standards, either with or without reservations, were classified as having a positive or 
negative effect on student outcomes if the findings were statistically significant. The number of studies that 
the WWC identified, screened, deemed eligible, and ultimately included as supporting evidence in the practice 
guide can be found in Figure B.1. 


FIGURE B.1. STUDIES IDENTIFIED, SCREENED, AND REVIEWED FOR PRACTICE GUIDE 


Total records identified 
(n=51 [ 


IDENTIFICATION 


Records screened in 
for topic relevance — 9 


(n=3 [ 


Records excluded 
(n=47,437) 


Records a for review Records ineligible for review 
—— 
(n=282) (n=3,542) 


Studies that meet WWC 
TN(@MOKS (@)\\ Standards without 


ELIGIBILITY 


SCREENING 


STATUS/ reservations (n=20) 


EVIDENCE RATING 


Studies that do not meet 
——*® WWC Standards or provide 
credible evidence (n=253) 


Studies that meet WWC 
Standards with reservations 
(n=7) 


Studies that do not meet 
WWC Standards but 
provide credible evidence 
(n=2) 
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Phase 3: Generating the Recommendations 


After the literature review, WWC staff convened a meeting with the panel to discuss the results. The panel 
considered details of 29 studies, including each one’s WWC evidence ratings, setting, sample, and findings. 
Adding their expert guidance, the panel produced five recommendations on how to use technology to support 
postsecondary student learning. The panel also assigned a level of evidence rating to each recommendation. The 
ratings capture both the confidence of the panel in the effectiveness of the recommendation and the strength 
of the evidence for the effect of the practice on postsecondary student outcomes (see Appendix A for the IES 
ratings scheme). 


The five recommendations in this practice guide are supported by 27 studies that meet WWC group design 
standards with or without reservations and by two additional studies that did not meet WWC group design 
standards. Findings from those two studies were included in Recommendation 1 because the panel determined 
that in supporting that recommendation, they added value beyond the studies that did meet standards. 
Specifically, each of the two studies offers a “direct test of the recommendation’ (on this point, see “Relationship 
Between Research and the Recommendations’ in Table A.1 of Appendix A). 


Phase 4: Drafting the Practice Guide 


WWC staff worked in conjunction with the panel to draft the Practice Guide. This included drafting the 
following for each of the five recommendations: evidence summary, implementation guidance, panel advice 
on overcoming potential obstacles, and tools and resources. The Practice Guide underwent several rounds 

of internal review, which included review from postsecondary education advisors, as well as IES external peer 
review (as described in Appendix A). 
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Appendix C: Rationale for Evidence Ratings 


The level of evidence assigned to each recommendation is based on expert guidance from the panel members 
and the findings from studies that examined the effectiveness of recommended practices. As noted in the 
introduction to this practice guide, rigorous research on the effects of technological interventions on student 
outcomes is rather limited. More rigorous research on new technologies and how best to support instructors’and 
administrators’ uses of technology is needed. 


For this practice guide, study findings in an outcome domain are classified as having one of the following effects:* 


e Statistically significant positive effect. These findings indicate a positive impact of the intervention on the 
outcome, statistically significant at p < .05. 


e Indeterminate effect. These findings do not indicate a directional impact of the intervention. They are not 
statistically significant. 


e Statistically significant negative effect. These findings indicate a negative impact of the intervention on 
the outcome, statistically significant at p < .05. 


Findings reported in studies that meet WWC group design standards could have been adjusted by the WWC in 
two scenarios: (1) the author reported multiple comparisons within a given outcome domain and did not adjust 
the critical p-value when determining significance; or (2) the unit of assignment in an randomized controlled 
trial (RCT) was different from the unit of analysis, such as when course sections are randomly assigned to 
conditions, individual student test scores are analyzed, and the clustering of students within course sections is 
not accounted for in the analysis model. In scenario (1), the WWC applied a Benjamini-Hochberg correction. In 
scenario (2), the WWC applied a clustering correction. For more information about the corrections applied by the 
WWC, see the What Works Clearinghouse Procedures and Standards Handbook, Version 3.0. 


ELIGIBLE POPULATIONS. The recommendations in this guide are primarily intended for instructors, 
administrators, and other staff who work with postsecondary students at 2-year and 4-year institutions. This 

can include undergraduate, graduate, and professional students, as well as students in technical or vocational 
education. Across the 29 studies that support the recommendations in this practice guide, all have samples that 
include undergraduate students and two have samples that include graduate students. 


ELIGIBLE OUTCOMES. The study outcomes eligible for inclusion in this practice guide fall into one of the 
following six domains: 


1. Academic achievement refers to the extent to which a student masters academic content. Eligible measures 
of academic achievement arise naturally from student educational experiences; those measures include final 
grades, grade point average, the ratio of college-level courses passed vs. failed, and unit exam scores. 


2. College attendance refers to outcomes that measure attendance rates or absenteeism. 


3. Credit accumulation and persistence refers to progress toward the completion of a degree, certificate, or 
program. Outcomes can include the number of college-level credits earned or enrolled (accumulation) and 
whether the student did or did not enroll the next semester (persistence). 


4. Attainment of a degree, certificate, or credential refers to graduation from or the otherwise successful 
completion of a degree, certificate, or credential. 


’ Statistically significant findings are those unlikely to have occurred by chance. 
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5. Post-college employment and income refers to outcomes related to employment after the 
postsecondary experience, including employment and income. 


6. Student engagement and motivation refers to outcomes related to motivation, intrinsic and extrinsic; 
engagement; and related constructs such as goal orientation and self-regulation. 


Findings in three of the domains above are captured in the 29 studies that support the recommendations in 
this practice guide: academic achievement outcomes are addressed in 27 studies, credit accumulation and 
persistence outcomes are addressed in four studies, and student engagement and motivation outcomes are 
addressed in one study. None of the 29 studies that meet WWC standards captured findings in any of the other 
three domains (attendance, attainment, or employment). 


CLASSIFYING THE INTERVENTION AND COMPARISON CONDITIONS. Some studies that support the 
recommendations in this practice guide evaluated multiple treatment conditions, or compared a treatment 
condition to a comparison condition where technology supports were also offered. In the tables included 

in this appendix, the two columns “Intervention Condition” and “Comparison Condition” describe what 
supports were available to students in each condition. This information is intended to help readers understand 
what comparisons were made in the studies that led the panel to determine that the findings support the 
recommendations. For example, intervention conditions that are described as technologies that provide low- 
cost access to course content could have academic achievement domain outcomes equal to outcomes of 
students in a traditional face-to-face comparison condition. In such a case, the finding can be interpreted as 
positive, as the technology is expanding access to education without sacrificing academic performance. 


STUDY DETAILS. Details about the participants, setting, intervention and comparison conditions,* outcomes, 
and effect sizes for the 29 included studies are presented in the tables that follow. Each table is preceded by a 
justification for why the findings, when considered collectively across studies supporting that recommendation, 
lead to its assigned evidence rating. Inside each table, studies that meet WWC group design standards are listed 
alphabetically by first author; studies that do not meet WWC group design standards are presented at the end. 
For studies that included multiple outcomes in a domain, reported effect sizes and statistical significance are 
for the domain and calculated as described in the WWC Procedures and Standards Handbook, Version 3.0 (pp. 
27-29). A domain average across all studies that contribute to a recommendation is presented at the end of 
each table. These averages were generated by a fixed-effect meta-analysis model using inverse-variance weights. 
One feature of this approach is that effects generated from larger sample sizes have more weight than those 
generated from smaller samples (see Hedges & Vevea, 1998). 


*The description of the intervention condition includes information about the duration of the intervention. The WWC does not take into account length of 
exposure to an intervention when summarizing findings. 
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Recommendation 1. Use communication and 
collaboration tools to increase interaction among 
students and between students and instructors. 


Rationale for Level of Evidence: Minimal 


WWC staff and the expert panel assigned a minimal level of evidence based on three studies of the effectiveness of 
communication tools designed to foster collaboration and build community. One of the studies°® meets WWC group 
design standards without reservations, and the two other studies*’ do not meet WWC group design standards. 


Evidence from all three studies provided a direct test of the recommendation. Junco and colleagues (2011) 
examined the effects of a Twitter-based intervention encouraging interaction around educationally relevant 
topics on the achievement of undergraduate pre-health majors. The study conducted by Fagioli and colleagues 
(2015) did not meet WWC standards because baseline equivalence was not established. Still, it had a very large 
sample across seven community colleges and used a rigorous matching approach to form intervention and 
comparison groups.°® It examined the impact of Facebook-based Schools App on student achievement. Nalbone 
and colleagues (2016) used a sample of approximately 1,000 students at one university, but the WWC could not 
assess whether the intervention and comparison groups were equivalent at baseline. Those authors examined 
the effects of a Facebook-based social network on student retention. 


This evidence suggests positive effects of communication and collaboration tools on academic success 
outcomes, although some indeterminate effects were observed. Academic achievement was assessed in two 

of the three studies.°? Junco and colleagues (2011) reported that the end-of-semester GPA for the intervention 
students was 2.79 versus 2.28 for the comparison group students, a statistically significant difference. Fagioli and 
colleagues (2015) also reported a statistically significant impact on end-of-term GPA. Persistence was assessed in 
two of the three studies;°° both studies reported statistically significant positive impacts on persistence in favor 
of the intervention groups. Student engagement was reported in one study,°*' which found that students in the 
Twitter collaboration and community-building intervention exhibited significantly higher engagement than 
students in the comparison group. 


The panel determined all three studies involved direct tests of the recommendation, were conducted in college 
course settings, and involved a large number of students across nine postsecondary institutions. However, only 
one of the three studies meets WWC standards. Therefore, the panel has assigned a minimal level of evidence for 
this recommendation. This rating is supported by strength of the evidence according to the following criteria: 


e Consistency of Effects on Relevant Outcomes. Two studies related to this recommendation found 
positive and statistically significant effects on measures of academic achievement. Two studies found 
positive and statistically significant effects on credit accumulation and persistence. One study found 
positive and statistically significant effects on student engagement and motivation.” No studies found 
negative effects on any outcome. 


e Internal Validity of Supporting Evidence. One study is a randomized controlled trial (RCT) with low attrition 
that meets WWC group design standards without reservations.© Two other studies that contribute evidence 
are quasi-experimental designs (QEDs) that do not meet WWC group design standards. One study that 
does not meet WWC group design standards uses propensity score matching to form intervention and 
comparison groups.%” 
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e Relationship Between the Evidence and Recommendation 1. The study interventions consist of 
communication and collaboration tools, particularly social networking tools, designed to facilitate 
interaction and build a sense of community among students and faculty. Evidence from all three studies 
used to support Recommendation 1 is considered to be a direct test of the recommendation. 


e External Validity of Supporting Evidence. The student samples across all studies include 144,458 
students in nine postsecondary institutions. Both 2- and 4-year colleges are represented in the study 
samples. The interventions occurred during the academic year and were tested in natural classroom and 
campus settings. 


Table C.1. Studies providing evidence for Recommendation 1: Use communication and collaboration tools to 
increase interaction among students and between students and instructors 


Outcome 
Domain and WWC 


Calculated Effect 


Participants Setting Talesvav(valarelam@eyareliacoya Comparison Condition Size 
Fagioli,Rios- 143,307 7 Schools App Users: Students in Nonmembers of Schools Academic 
Aguilar, & undergraduates community the intervention condition joined App: The comparison achievement: 
Deil-Amen colleges in the Schools APP, a Facebook- condition is the students Aetreieee Gok wee 
(2015) 7 states (AR, based application for use only at the study colleges 0.04 points higher 
OED AZ,CA,OH, by invited students, faculty, and during the same terms hee AGnmembere 
NY,TX,and staff at a specific college. The who chose not to join the 
Does not WY) app is designed to facilitate and Schools App. Passive poses GPA 
meet WWC manage social engagement for was 0.02 points 
group design college students, allowing them higher than 
standards to get to know other students nonmembers* 
at the college, organize social Credit accumulation 
activities offline, access campus and persistence: 


information, ask questions, and 
seek advice. The intervention 
condition is subdivided into 


Active users: 1.28 
times more likely 
to persist than 


active users (those who posted a 
or commented on the app, liked nonmembers 
other users’ comments, or joined Passive users: 1.35 
offered meet-ups); passive users times more likely 
(those who spent more than to persist than 

a minute on the app but did nonmembers” 
not complete any of the active 

activities); and inactive members 

(those who did not spend any 


time on the app). 


(Intervention Duration: 2 semesters) 
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Participants Setting 
Junco, 118 1 US college 
Heiberger,& undergraduate 
Loken (2011) — pre-health 
RCT professional 
majors 
Meets WWC 
group design 
standards 
without 
reservations 
Nalbone et 1,033 1 university 
al. (2016) undergraduates in the 
Midwest 
QED 
Does not 
meet WWC 
group design 
standards 


Domain Averages for Recommendation 1 


Tatc-yavc-vararoyam@eyaveliateyay 


Twitter: Twitter was used to 
continue class discussions, 

allow students to ask questions, 
remind students of assignments 
and campus events, provide 
information on academic and 
personal support services, and 
organize service learning projects 
and study groups. 


(Intervention Duration: 14 weeks) 


Facebook Group Participants: 


Comparison Condition 


Business as Usual 
Instruction with Ning: The 
comparison sections were 
taught using business- 
as-usual instructional 
methods with the Ning 
social networking site 

for the usual learning 
management system. 
nstructors in the 
comparison sections used 
ing to provide students 
the same information that 
was provided to students 
in the intervention sections 
via Twitter. Faculty-student 
interaction on Ning was 
minimal compared to the 
intervention condition. 


Outcome 
DYeoynaY-liaw-vavem Aue 
Calculated Effect 
Size 


Academic 
achievement: 
g = +0.52* 


Student 
engagement: 
g = +0.64* 


onparticipants in the 


The intervention group consisted 
of first-year students who were 
part of the university's Center 

for Student Achievement (CSA), 
were enrolled in a first-year 
experience course for CSA 
students only, and opted to 
participate in both (1) a Facebook 
group for only members of their 
first-year experience course and 
(2) a Facebook group for all first- 
year students in the intervention 
group. Two cohorts of students 
participated in the study. 


(Intervention Duration: 2 years) 


(studies that meet WWC group design standards only) 


Facebook Group: The 
comparison group 


consisted of first-year 
students who were not a 
part of the CSA and were 
enrolled in a non-CSA first- 
year experience course 
with a professor who 
volunteered to be part of 
the study. These students 
did not receive the offer to 
join the Facebook groups. 


Credit accumulation 
and persistence: 
g=+1.05* 


Academic 
achievement: 
g =+0.52* 


Student 
engagement: 
g=+0.64* 


@ There was insufficient information reported in Fagioli et al. (2015) to generate effect sizes for the academic achievement and credit accumulation and persistence 
domains. In the table, coefficients from the analysis models are presented. 


* = statistically significant at the .05 level. 
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Recommendation 2. Use varied, personalized, and 
readily available digital resources to design and deliver 
instructional content. 


Rationale for Level of Evidence: Moderate 


WWC staff and the panel assigned a moderate level of evidence based on 16 studies (see Table C.2). Eleven 
studies meet WWC group design standards without reservations. Five studies meet WWC group design 
standards with reservations.” 


Of the 16 studies, seven are relevant to Recommendation 2a, exploring redesigning instruction using 
technology.”' All seven studies are a direct test of the recommendation; the studies compared blended, flipped, 
or online course sections to sections of the same course taught in a traditional face-to-face format. In one of the 
studies,”” the blended section was accelerated, whereas the traditional section took a full semester. Results of a 
meta-analysis of these seven studies indicate that students in the blended, flipped, or online sections 

performed, on average, as well on academic achievement measures as their counterparts in traditional sections 
(g =-0.01, p =.80), indicating that the delivery of portions of course content online is not detrimental to 
academic performance. The panel interprets this as a positive finding. If students in blended learning courses can 
achieve similar academic progress as students in traditional lecture courses, then blended approaches can be 
delivered to a broader audience, with less classroom time, and at a lower cost. 


In one study” related to Recommendation 2a, the impact of blended, accelerated instruction on credit 
accumulation and persistence was positive and statistically significant. Eighty-seven (87) percent of students in 
the blended sections completed the course, compared to 82 percent of students in the face-to-face sections. 
Furthermore, 80 percent of students in the blended sections passed the course, compared to 76 percent 

of students in the face-to-face sections. In another study,” the impact of a flipped classroom format on the 
comprehensive assessment in statistics course measure was positive and statistically significant. 


Of the 16 studies, nine provide support for Recommendation 2b, exploring the effects of various types of 
enhancements to online resources or digital resources for delivering instructional content such as online review 
tools, podcasts, and visualization aids.” All studies are a direct test of the recommendation, and all examined 
outcomes in the achievement domain. The outcome measures in most of these studies were narrow knowledge 
tests related to the content delivered through the digital resource; one study’® examined the impact of a 
redesigned course on final course grades. Results of a meta-analysis of these nine studies indicate a positive 

and statistically significant effect of the digital resources (g = 0.35, p < .01). This effect size means that we would 
expect a comparison group student who earned the median score (50th percentile) on a knowledge test to have 
scored in the 62nd percentile if she had been in the intervention group. 


Each study supporting this recommendation has high internal validity. The collection of studies demonstrate 
high external validity and relevance to a range of postsecondary settings. Nearly all of the studies find positive 
or neutral effects on academic achievement. Nine of the studies supporting this recommendation provide a 
direct test of digital resources for delivering instructional content, whereas the other seven compared blended, 
flipped, or online courses to traditional face-to-face courses. The strong internal and external validity of the 
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supporting studies, and the preponderance of positive effects among studies that provide a direct test of the 
recommendation, has led the panel to assign a moderate level of evidence to this recommendation. This rating is 
supported by strength of the evidence according to the following criteria: 


e Consistency of Effects on Relevant Outcomes. The studies supporting the recommendation found 
positive, indeterminate, and negative effects in the academic achievement domain. Six studies found 
positive effects on outcomes in the academic achievement domain.” Nine studies found indeterminate 
effects in this domain.’”® One study found negative effects on academic achievement.”” One of the 
studies that found indeterminate effects on academic achievement also found positive effects on credit 
accumulation and persistence.” 


e Internal Validity of Supporting Evidence. The studies supporting this recommendation have strong 
internal validity. All 11 studies that meet WWC group design standards without reservations were RCTs with 
low sample attrition.*' The five studies that meet WWC group design standards with reservations 
used QEDs. 


e Relationship Between the Evidence and Recommendation 2. The studies supporting this 
recommendation examine interventions that are closely aligned with at least one element of 
Recommendation 2. Six of the studies compare blended courses to traditional face-to-face courses. One of 
the studies compares online course delivery to traditional face-to-face. The other nine studies test a variety 
of digital resources for delivering instructional content. 


e External Validity of Supporting Evidence. The 16 studies supporting this recommendation compared the 
recommended practices to traditional classroom instructional techniques. Across all studies, the samples 
consisted of 3,380 undergraduate students at postsecondary institutions across the United States. The 
length of the interventions varied, ranging from a 12-minute podcast to a full-semester course. 
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Table C.2. Studies providing evidence for Recommendation 2: Use varied, personalized, and readily available 
digital resources to design and deliver instructional content 


Outcome 
Domain and WWC 


Calculated Effect 
Participants Setting Tal <=lav(-lahColam@olarelia (ola) Comparison Condition Size 


RECOMMENDATION 2A 


Banks (2004) 157 1 regionally Hybrid Instruction (Accelerated Traditional Classroom Academic 
OED undergraduate accredited Format): The intervention Instruction (Accelerated achievement: 
and graduate institution condition incorporated Format): The comparison Gi O05 
Meets WWC students serving asynchronous delivery of weekly group received instruction 
group design students in course module content (assigned in an accelerated format 
standards with 4 states (AZ, readings) using Microsoft during weekly 4-hour in- 
reservations CA,CO,and Outlook Express. Students in person workshops in the 
UT) the intervention condition evening. 

responded to discussion 

questions posted to newsgroups 

by the instructor and other 

students. Communication among 

students and between students 

and the instructor was primarily 

asynchronous, through email. 

Two-thirds of the course 

content was covered online 

and one-third was covered in a 

traditional classroom. 


(Intervention Duration: 1 
semester) 
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Participants 


Bowen, 605 
Chingos, undergraduate 
Lack, & students 
Nygren 

(2014) 

Additional 

source: 

Bowen, 

Chingos, 

Lack, & 

Nygren 

(2012) 


CT 


Meets WWC 
group design 
standards 
without 
reservations 


Coates, 126 students 
Humphreys, 

Kane, & 

Vachris 


(2004) 
QED 


Meets WWC 
group design 
standards with 
reservations 
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Setting 
6 public 


universities 


(NY and 
MD) 


3 public 
universities 
(NY, MD, 


and VA) 


semester) 


Tatc-yavc-varacoyam@eyaveliareyay 


nteractive Learning Online (ILO): 


Comparison Condition 


Face-to-Face Instruction: 


Students assigned to the hybrid 
statistics courses participated in 
LO, which included machine- 
guided instruction and face- 
to-face instruction each week. 
These sections were delivered 
in a hybrid mode, in which 
most of the instruction was 
delivered through interactive 
online materials, but the online 
instruction was supplemented 
by a weekly 1-hour face-to 
face session, so that students 
could ask questions or be given 
targeted assistance by the 
instructor. The ILO intervention 
included textual explanations of 
concepts, worked examples, and 
practice problems. Students were 
also required to manipulate data 
using statistical software. 


(Intervention Duration: 1 


Online Instruction: Students 

in the intervention condition 
took a principles of economics 
course, either microeconomics or 
macroeconomics, entirely online 
Each participating university 
used a different textbook and 
mode of online delivery. Either 
asynchronous or synchronous 
communication discussion 
forums were used. All three 
universities use electronic tests 
and exams and allow for extra 
credit in the online courses. 


(Intervention Duration: 1 
semester, 


Students assigned to 

the traditional statistics 
courses were taught 
introductory statistics as it 
is usually offered at their 
institutions, with face-to- 
face instruction. 


Business As Usual Lecture: 
Comparison students 

in the one principles of 
economics course at 
each institution involved 
face-to-face business as 
usual lectures using the 


same content as the online 


course. 
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Outcome 
DYeynaY-liaw-Vavem Aue 
Calculated Effect 
Size 


Academic 
achievement: 
g = +0.09 


Credit accumulation 
and persistence: 
g = +0.19* 


Academic 
achievement: 
g=-0.39* 
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Participants Setting 


Lovett, 61 
Meyer, & undergraduate (PA) 
Thille (2008) = students 


Additional 
Source: 
Lovett, 
Meyer, & 
Thille (2010) 
RCT 

Meets WWC 
group design 
standards 


without 
reservations 


1 US 
university 


Overmyer 301 
(2014) undergraduate 
QED students 


Meets WWC 
group design 
standards with 
reservations 
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Open Learning Initiative (OLI), 


(eoyin) ey-Ta oem @eoyaveliacoya) 


Traditional Course: 


Accelerated Course: Students 
enrolled in an interactive online 
statistics course. Students in 

the OLI course acquired most 

of the course content online, 
meeting with an instructor 
approximately two times a 
week for 50-minute sessions to 
ask questions and review more 
challenging material. The course 
was completed in an accelerated 
format (8 weeks instead of 15). 


(Intervention Duration: 8 weeks) 


Flipped Classroom: Students 
enrolled in the flipped classroom 


sections of a college algebra 
course watched recorded video 
lessons outside of class time. 
Instructors created 30 video 
lessons that had an average 
runtime of 20 minutes. Instructors 
also made solution videos for 
exams and posted them online 
after tests were returned to 
students rather than going 

over exams in class. Students 
completed basic homework 
questions (multiple-choice 
questions focused on concepts 
and vocabulary) outside of 

class after watching the lecture 
videos. During class time, 
students worked on additional 
homework questions involving 
formulas and applications. Some 
sections of the flipped classroom 
included collaborative group 
work, inquiry-based learning, and 
active whole-class discussions 
during class time. 


(Intervention Duration: 14 weeks) 


Students received 
traditional instruction in 
a 15-week introductory 
statistics course. The 
content was the same 
as that delivered in the 
intervention condition. 


Traditional Classroom: 
Students enrolled in the 
traditional sections of a 
college algebra course 
watched live lectures 
during class time. All 
homework problems were 
assigned to be completed 
outside of class time. The 
homework problems were 
exactly the same for the 
flipped classroom model 
and for the traditional 
classroom model. 
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Outcome 
DYeoynaY-liaw-vavem Aue 
Calculated Effect 
Size 


Academic 
achievement: 
g = +0.55* 


Academic 
achievement: 
g=+0.11 
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Outcome 
Domain and WWC 
Calculated Effect 


Participants Setting Tale=dav(vald (elem @earel lacey Comparison Condition Size 

Schunn & 188 Large, Open Learning Initiative (OLI), Traditional Course: Academic 

Patchan undergraduate public Logic and Proofs Course: Students received achievement: 

(2009) students research Students enrolled in an traditional instruction in a g=—0:07 

QED institution interactive online logic and 13-week logic and proofs 

in the US proofs course. This course course. The content was 

Meets WWC combined online instruction with the same as that delivered 

group design a small amount of targeted face- _in the intervention 

standards with to-face instruction. condition. 

reservations ; 

Intervention Duration: 13 weeks) 

Yong, Levy,& 172 1 college Flipped Classroom: Students Traditional Classroom: Academic 

Lape (2015) undergraduate (CA) in the flipped sections of ost of class time was achievement: 

Additional students an introductory differential spent on lectures relying @g=—025 

ource: equations course were assigned on the same PowerPoint 

Lape (2016) to watch one to two 15-minute SI des and materials used 

ecture videos that presented n the flipped sections. 

RCT PowerPoint slides with “live Classes usually included at 

Meets WWC virtual ink annotations’ and east one practice problem 

group design audio narration by the professor. that students could work 

standards An online survey was given on independently or 

without at the end of each video to in small groups, during 

reservations determine whether students which time the professors 

watched the assigned videos. At = would walk around the 
the beginning of the class, the oom to assess students’ 
professor answered questions understanding of the 
about the videos. During material and answer 

the rest of the course time, questions. All homework 
students worked independently questions were completed 
or cooperatively on selected outside of class time. The 
homework problems, and duration of the course 
the professors interacted and amount of total class 
with students by asking and time mirrored the flipped 
answering questions. A portion sections. 

of class time was often reserved 

for working on mathematical 

modeling tasks in small groups. 

(Intervention Duration: 7 weeks) 

Domain Averages for Recommendation 2a Academic 
Achievement: 
g=-0.01 
Credit 
accumulation and 
persistence: 
g=+0.19* 
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Outcome 
Domain and WWC 
Calculated Effect 


Participants Setting aecyaUclalacelm@eorareliaceyay Comparison Condition Size 
RECOMMENDATION 2B 
Bannan- 137 l university Visual Summary with Text and Graphics Only: In = Academic 
Ritland & undergraduate in eastern anipulation: Instruction on the comparison condition, achievement: 
Grabowski students in three US heart content is provided via students received an G—.029 
(2002) statistics courses webpages using hypertext and online instruction module 
RCT images that can be manipulated. on heart content via text 
Students can click and drag and graphics only. 
Meets WWC images in the web-pages and are 
group design presented with text information 
standards that supports the visuals. 
without ; : 
(acon atiOns Intervention Duration: one lesson, 
ength of lesson unknown) 
Learner Manipulation: Instruction Academic 
on heart content is provided via achievement: 
webpages using hypertext and GeO 38 
images that can be manipulated. 
Students can click and drag 
images in the web-pages, but no 
text information that supports 
the visuals is presented. 
Intervention Duration: one lesson, 
ength of lesson unknown) 
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Outcome 
Domain and WWC 
(@:|aU ele-vom sive 


Participants Setting TahecyaVclalacelam @eorareliaceyay (@oyin) ex-Taexolameyave! i aco) s MS) 74-1 

Chou (2009) 62 l university Visualized Concept Maps: Hypertext Only: In the Academic 
Additional undergraduate (PA) Instruction on heart content comparison condition, achievement: 
eee students is provided via three to five students received an G=-aksoe 
Chou (2013) webpages using hypertext and online instruction module 

visualized concept maps. In this on heart content via 
RCT condition, hypertext is presented — hypertext only. 
Meets WWC along with 19 concept maps 
group design that relate to the instructional 
standards content. The concept maps 
without comprise oval text boxes that 
reservations represent concepts and arrows 

that represent relationships 

between concepts. When 

students move their mouse 

pointer over the ovals in the 

concept map, related images 

are presented (using Flash 

animation) on the screen that 

einforce the concept. 

Intervention Duration: 25 

minutes) 
Couzin 420 1 statewide Online Course Modified to Unmodified Online Academic 
(2016) undergraduate community Incorporate Universal Design Course: Inthe comparison — achievement: 
RCT students college for Learning Principles: The condition, students took Gi—— 0105 

system intervention students took an the original unmodified 

Meets WWC online writing course that was version of the online 
group design modified to implement Universal — writing course. 
standards Design for Learning (UDL) 
without principles focusing on multiple 
reservations means of representation, 

engagement, and expression. 

These principles were applied to 

numerous elements of the course 

including session checklists, web- 

based games, practice quizzes, 

rubrics, additional videos, and 

infographics, among others. 

(Intervention Duration: 16 weeks) 
Evans, Yaron, 45 1 university | OLI Stoichiometry Review Text-Only Study Guide: Academic 
& Leinhardt undergraduate (PA) Course: This course includes an Students assigned to the achievement: 
(2008) students overarching real-world story, an comparison group were G= 063" 
Additional explorato y virtual laboratory, a given a text-only study 
source: variety of practice contexts, and guide. The study guide 
Evans (2007) immediate feedback on actions covers the same content 

and submitted responses. as the online review course 
ie (Intervention Duration: 20-25 but does not include the ; 
Meets WWC hours) overarching story, dynamic 
group design interface, or feedback. 
standards The review guide takes an 
without estimated 12 to 15 hours 
reservations to complete. 
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Outcome 
Domain and WWC 
Calculated Effect 


Participants Setting aecyaUclalacelam@eovavel lacey) (oyna) ex-Taexolam @oyave! aco) s MES) 74-1 
Fratangeli 84 l university | Podcast-Based Lesson: Regular Lecture-Based Academic 
(2009) undergraduate in the Students in an introductory Lesson: Students in the achievement: 
RCT students Mid-Atlantic communications class were comparison group were G=—0s5) 
given a podcast-based lesson. given a regular lecture- 
ae ee (Intervention Duration: 1 lesson) based lesson. 
standards 
without 
reservations 
Galbraith Oy l university — Instructor-Directed Multimedia Student-Directed Academic 
(2014) undergraduate (ID) Format and Sequencing: Multimedia Format and achievement: 
RCT students Students completed an online Sequencing; Students G= 0508 
farnily finances learning unit self-directed the LO 
Meets WWC total viewing time of 152 sequence and self-selected 
group design minutes). The unit consisted of the multimedia format 
standards 12 Learning Objects (LOs). The for each LO. The LOs were 
without instructor sequenced the LOs available in a variety 
reservations and picked one multimedia of multimedia formats 
format for each LO for the including text, PowerPoint, 
students in this condition. audio, and video. 
The content and assignments 
were identical to those in the 
comparison condition. 
(Intervention Duration: 1 week) 
Kennedy et 142 1 university Content Acquisition Podcast Reading a Practitioner- Academic 
al. (2013) undergraduate (VA) CAP) with Lecture: Students in Oriented Article: Students achievement: 
RCT students the intervention group watched in the comparison group Ga 
a 12-minute Content Acquisition — read a practitioner- 
Meets WWC Podcast (CAP) on phonological oriented article on the 
group design awareness. The CAP contained same topic. This article's 
standards three sections: (1) What is content aligned to the 
without phonological awareness? (2) content of the CAP and 
reservations Why is phonological awareness students were instructed 
necessary? (3) What are effective to take notes at their 
ways of teaching phonological discretion. Students were 
awareness to students? When all —_ allowed to take as much 
students completed their work time as they needed to 
in the treatment and comparison __ read the article. 
conditions, students received the 
regularly scheduled lecture for 
approximately 90 minutes. 
(Intervention Duration: 12 minutes) 
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Outcome 


Domain and WWC 
(@:|aU ele-vom sive 


Participants Setting TahecyaUclalacelam@eorareliaceyay (oyna) ex-Taxolam eyave! dco) s MS) 74-1 
Rhoads 146 l university Podcasted Reviews of Course n-Class Oral Reviews: Academic 
(2010) undergraduate in the Material: The intervention Control classrooms achievement: 
RCT students Midwest condition in this study involved 5- received traditional in-class g=+0.13 
to 7-minute podcasted reviews of _ oral reviews prior to taking 
Meets WWC course material, including audio the exams. Otherwise, the 
gay design with pictures and slide images, comparison classrooms 
standards ‘an prior to course exams. The course — functioned the same and 
teas Ons was focused on health concepts had the same content 
for future teachers; content was and PowerPoint lectures, 
delivered via lectures andin- and — assignments, and activities. 
out-of-class projects. Professors 
allowed for an in-class review 
period the day before an exam. 
In the intervention condition, 
four podcasts on acute infections 
and chronic conditions including 
asthma, diabetes, and seizure 
disorders were created by the 
author. The author used the 
professors’ lecture slides to create 
the podcasts, which were vetted 
by the professors for accuracy. 
Students came to the regularly 
scheduled class to view and listen 
to the podcasts. 
(Intervention Duration: 1 
semester) 
Trevisan, Oki, Experiment 1: 6 US Time Compressed Animated Traditional Lecture on Video: Academic 
& Senger 364 universities Delivery (TCAD 3D): Students Students in the comparison achievement: 
(2010) undergraduate in the intervention condition condition viewed a traditional g = +1,09* 
RCT students viewed a Time Compressed lecture on video. The lecture 
Animated Delivery (TCAD 3D) consisted of a PowerPoint 
Meets WWC teaching follicular dynamics. The presentation with intermittent 
group design digital animated presentation, Views of the instructor, 
standards which was explained by voice- naive pvolecicisiome 
without over, was projected on a large Pybionieial tgs seen io) 
reservations screen for classroom viewing. esa ne Meare) 
of the video-lecture was 
(Intervention Duration: 17 minutes) — approximately 34 minutes. 
Experiment 6 US Time Compressed Animated Traditional Lecture on Academic 
De DR universities Delivery (TCAD 3D): The TCAD Video: The length of this achievement: 
undergraduate 3D content was more in-depth video was approximately G=—02on 
students compared to Experiment 1 and 23 minutes. 
described differential hormone 
regulation. 
(Intervention Duration: 17 minutes) 
Domain Averages for Recommendation 2b Academic 
Achievement: 
g=+0.35* 


* = statistically significant at the .05 level. 
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Recommendation 3. Incorporate technology that models 
and fosters self-regulated learning strategies. 


Rationale for Level of Evidence: Moderate 


WWC staff and the panel assigned a moderate level of evidence based on four studies that examined the 
effectiveness of various strategies to foster or activate student self-regulation (see Table C.3). All four studies 
meet WWC group design standards without reservations.” 


All four studies provide a direct test of the recommendation. Sullivan (2016) investigated the impact of text 
and email course activity reminders in a blended course. Delen and colleagues (2014) studied an interactive 
learning environment that included supporting elements such as note taking, additional resources, and 
practice questions. Lin and Lehman (1999) examined the use of reason-based, rule-based, and emotion-based 
justification prompts in a computer simulation activity. Kauffman and colleagues (2011) explored combinations 
of structured matrix or outline note taking with self-monitoring prompts in a web-based learning module. 


All studies related to this recommendation reported positive effects for outcomes in at least one outcome 
domain. Student academic achievement was assessed in all four studies.®° Results of a meta-analysis indicated 
a statistically significant positive effect on their achievement in favor of the students receiving self-regulation- 
promoting technologies (g = 0.66, p < .05). Student persistence was reported in one study,*” which was 
determined by the WWC to be an indeterminate effect (g = 0.31, p =.15). 


The panel believes that the research supports a moderate level of evidence for this recommendation. The strong 
internal validity of the four studies, the largely consistent effects on student academic achievement, and the 
range of self-regulation-promoting technologies tested all point to the benefits and broad applicability of the 
technologies and interventions tested. This rating is supported by strength of the evidence according to the 
following criteria: 


e Consistency of Effects on Relevant Outcomes. The evidence for this recommendation focuses primarily 
on student achievement outcomes. Three studies related to this recommendation demonstrated consistent 
positive effects on academic achievement outcomes, which were measured by instructor- or researcher- 
created assessments. Two of these three studies demonstrated statistically significant effects when testing 
more robust scaffolds of self-regulated learning.® For example, matrix note-taking methods yielded more 
positive effects than the outline note-taking method, and reason-justification metacognitive prompts were 
more effective than rule-based prompts. 


e Internal Validity of Supporting Evidence. The studies supporting this recommendation have strong 
internal validity. All four studies were RCTs with low sample attrition that meet WWC group design 
standards without reservations. 


e Relationship Between the Evidence and Recommendation 3. The studies supporting this 
recommendation examined interventions that are closely aligned with the recommendation. Each provided 
a direct test of the effects of scaffolds designed to support students’ use of self-regulated learning strategies. 
However, there are many self-regulation-promoting applications that might be used to support this 
recommendation that have not been fully tested with rigorous research. 


e External Validity of Supporting Evidence. The studies supporting this recommendation have moderate 
external validity. Two interventions were delivered in courses and took place throughout an entire semester 
or school year,” whereas 10 were shorter in duration and delivered in lab settings.*' The four studies had 
small student samples, ranging in size from 20 to 88 students and totaling 291. 
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Table C.3. Studies providing evidence for Recommendation 3: Incorporate technology that models and 


fosters self-regulated learning strategies 


Participants 


Setting 


Tatc-yavc-varacoyam@eyaveliareyay 


Comparison Condition 


Outcome 
DYeoynaY-liaw-Vavem Aue 
Calculated Effect 
Size 


Delen, Liew, 
& Willson 
(2014) 


Additional 
source: 
Delen (2013) 


RCT 


Meets WWC 
group design 
standards 
without 
reservations 


80 1 college 
undergraduate (TX) 

and graduate 

students 


Kauffman, 
Zhao, & Yang 
(2011) 


RCT 


Meets WWC 
group design 
standards 
without 
reservations 
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Experiment 1: 1 college 
20 in the 
undergraduate Midwest 
students 


Online Video-Based Interactive 


Non-interactive Learning 


Learning Environment: The 
learning environment included 
a video viewer, interactive 
notes, practice questions, 

and additional resources that 
students could choose to use to 


learn about renewab 
The environment incl 
“control buttons” that 


e energy. 
uded 
allowed 


students to navigate 


the content 


and self-direct their learning. 


(Intervention Duration: 16 
minutes) 


Structured Online Note Taking, 


Environment: Students 
were given the same 
overall learning goal, 
content, and scaffolds 

as in the online video- 
based interactive learning 
environment. However, 
they were not able to self- 
direct their learning. 


Academic 
achievement: 
g=+1.11* 


Conventional Note Taking: 


atrix Method: Students 

took notes on a 2,000-word 
passage about wildcats. The 
matrix method organized the 
information students needed to 
collect into a two-dimensional 
table, with labels on the rows and 
columns and text boxes in the 
cells. 


(Intervention Duration: 1 
laboratory session) 


Structured Online Note 

Taking, Outline Method: The 
outline method organized the 
information students needed 
to collect into three hierarchical 
levels with labeled text boxes. 


(Intervention Duration: 1 
laboratory session) 


Students took notes on 

a 2,000-word passage 
about wildcats. The 
conventional method listed 
the information students 
needed to collect above a 
single text box lacking any 
structure or labeling. 


XATIONALE FOR EVIDENCE RATINGS 


Academic 
achievement: 
g = +0.96* 


Academic 
achievement: 
g=-0.24 
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Outcome 
Domain and WWC 
Calculated Effect 


Participants Setting aecyaUclala cele @eovareliaceyay Comparison Condition Size 
Kauffman, Experiment 2: 1 college Structured Online Note Taking, Conventional Note Taking: | Academic 
Zhao,&Yang 39 inthe Matrix Method Plus Self- Students took notes on achievement: 
(2011) undergraduate Midwest Monitoring Prompts: Students a 3,500-word text on gG=+2.13* 
RCT students took notes on a 3,500-word text educational measurement 
on educational measurement that that was divided into three 
Meets wwe was divided into three webpages. | webpages. 
group design Students saw a self-monitoring 
standards prompt at the bottom of each 
without webpage that encouraged them 
reservations to review their notes before 
moving on to the next page. 
(continued) g pag 
Intervention Duration: 1 
aboratory session) 
Structured Online Note Taking, Academic 
atrix Method Without Prompts achievement: 
— * 
Intervention Duration: 1 g= +1.36 
aboratory session) 
Structured Online Note Taking, Academic 
Outline Method Plus Self- achievement: 
onitoring Prompts g=+0.46 
Intervention Duration: 1 
aboratory session) 
Structured Online Note Taking, Academic 
Outline Method Without Self- achievement: 
Monitoring Prompts g=+0.35 
(Intervention Duration: 1 
laboratory session) 
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Participants Setting 


Lin & 
Lehman 
(1999) 


RCT 


Meets WWC 
group design 
standards 
without 
reservations 


88 
undergraduate 
students 


1 college 
inthe 
Midwest 


64 
undergraduate 
students 


Sullivan 
(2016) 


CT 


Meets WWC 
group design 
standards 
without 
reservations 


1 college in 
the South 


Domain Averages for Recommendation 3 


* = statistically significant at the .05 level. 
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Tatc-yavc-varacoyam@eyaveliareya) 


Reason Justification 
Metacognition Prompts: Students 
were prompted to provide 
reasons for their actions at three 
points during a biology lab 
simulation. 


(Intervention Duration: 1 biology 
laboratory session) 


Rule Based Metacognition 
Prompts: Students were 
prompted to explain rules or 
procedures. 


(Intervention Duration: 1 biology 
laboratory session) 


Emotion Focused Metacognition 
Prompts: Students were prompted 
to reflect on their feelings. 


(Intervention Duration: 1 biology 
laboratory session) 


Text Message-Based Course 


Comparison Condition 


No Metacognition Prompts 


No Metacognition Prompts 


No Metacognition Prompts 


Placebo Email Messages: 


Activity Reminders: These 
reminders directed students’ 


attention to what they needed 
to do at specific time points 
during the course, and reminded 
students regularly about the due 
dates and deadlines for tasks and 
assignments. The frequency and 
evel of detail in the reminders 
tapered out over the course of 
the semester. 


Intervention Duration: 1 semester) 


Email-Based Course Activity 


Placebo messages 
contained several practice 
questions before each quiz 
and the final exam. 


Placebo Email Messages 


Reminders 


Intervention Duration: 1 
semester) 


AL 


i 
| na 


FOR EVIDENCE 


RATINGS 


Outcome 
DYeoynaY-liaw-Vavem Aue 
Calculated Effect 
Size 


Academic 
achievement: 
g = +0.99* 


Academic 
achievement: 
g = +0.34 


Academic 
achievement: 
g=+0.23 


Academic 
achievement: 
g= +054 


Credit accumulation 
and persistence: 
g=+0.38 


Academic 
achievement: 
g = +0.57 


Credit accumulation 
and persistence: 
g = +0.24 


Academic 
achievement: 
g = +0.66* 


Credit 
accumulation and 
persistence: 
g=+0.31 
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Recommendation 4. Use technology to provide timely 
and targeted feedback on student performance. 


Rationale for Level of Evidence: Moderate 


WWC staff and the expert panel assigned a moderate level of evidence based on eight studies (see Table C.4). 
Five studies meet WWC group design standards without reservations.” Two studies meet WWC group design 
standards with reservations.” One study that tested multiple interventions reported on comparisons that 
meet group design standards without reservations and comparisons that meet group design standards with 
reservations.” 


Of the 10 interventions studied for this recommendation, six focused directly on technology-supported feedback 
in various forms. Four of the studies involved the use of student response systems in college classrooms.” One 
intervention was a student response system application in which students answered questions before class.” 
One intervention involved targeted, correct/incorrect feedback within a simulated learning task.?’ Results of a 
meta-analysis of these six studies evidenced a statistically significant positive impact on academic achievement 
in favor of technologies that employed timely and targeted feedback (g = 0.37, p < .01). 


The other four interventions studied involved online courses, a key component of which is embedded 
assessments with targeted and timely feedback for each instructional activity.” All four studies reported findings 
on student achievement, and one reported findings on credit accumulation and persistence. Results of a meta- 
analysis indicated a positive effect for the interactive courses on student achievement, but this effect is not 
Statistically significant (g = 0.11, p =.14). The impact on credit accumulation in the one study (Bowen, Chingos, 
Lack, & Nygren, 2014) that measured it was positive and statistically significant (g = 0.19, p < .05). The panel 
could not isolate the impact of feedback from the impact of other elements of the online courses such as high- 
quality text and graphics or practice activities; still they believe that the incorporation of assessment and tailored 
feedback into every learning activity within these courses is a critical part of these interventions. 


The strong internal and external validity of the supporting studies, the range of feedback technologies tested, 
and the largely consistent effects on student academic achievement have led the panel to assign a moderate 
level of evidence to this recommendation. This rating is supported by strength of the evidence according to the 
following criteria: 


e Consistency of Effects on Relevant Outcomes. As expected for technologies that are used mainly 
within single courses, rather than across students’ full schedules, the evidence focuses primarily on 
student achievement outcomes. One study, however, also examined the impact of feedback-facilitating 
technologies on another outcome of interest for this guide (i.e.,, effects on credit accumulation). The eight 
studies supporting the recommendation found both positive and indeterminate effects in the academic 
achievement domain. Five studies found statistically significant positive effects on outcomes in the 
academic achievement domain,” and five studies found indeterminate effects in this domain.'° One of the 
studies that found indeterminate effects on academic achievement also found positive effects on credit 
accumulation and persistence.'”' 


e Internal Validity of Supporting Evidence. The eight studies exhibit strong internal validity. Five studies 
were RCTs with low attrition that meet WWC group design standards without reservations.' One study was 
an RCT with high attrition that meets WWC group design standards with reservations.'” One study tested 
three interventions in different experiments, two of which were RCTs with low attrition that meet WWC group 
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design standards without reservations, and one of which is an RCT with high attrition that meets WWC group 
design standards with reservations.'* One study was a QED that meets WWC group design standards with 
reservations.’ 


e Relationship Between the Evidence and Recommendation 4. Four studies involved the use of either 
student response systems or other technologies to provide targeted feedback.'” These studies directly tested 
the recommendation. The Interactive Learning Online and Open Learning Initiative interventions, examined 
in four studies, include feedback components as part of more comprehensive intervention models that help 
students learn more productively by designing instruction and delivering content using varied, personalized, 
and accessible digital resources (see Recommendation 2).'° Though the evidence includes several types of 
technologies, such as student response systems and online courses with embedded feedback systems, there 
are other technologies for which no evidence that meets WWC standards was located. 


e External Validity of Supporting Evidence. The student samples across all studies included 1,356 students 
in 13 postsecondary institutions. Both 2- and 4-year colleges are represented in the study samples. The 
interventions occurred during the academic year and were tested in natural classroom and campus settings. 


Table C.4. Studies providing evidence for Recommendation 4: Use technology to provide timely and 
targeted feedback on student performance 


Outcome 
DYoynat-liameacem AN e 


Calculated Effect 


Participants Setting ahecyaUclalacelam@eovareliaceya) Comparison Condition Size 
Jones, Experiment 1: lcollegein  iClicker Classroom Response No CRS Academic 
Crandall, 49 the South System (CRS): Students used achievement: 
Vogler, & undergraduate the iClicker CRS to answer eight Gq=— One 
Robinson eden: multiple-choice questions for 
(2013) credit. 
RCT (Intervention Duration: 2 |ectures) 
en Experiment 2: 1collegein iClicker CRS for Credit iClicker CRS Not for Credit Academic 
eets : : 
group design 39 ene Sot (Intervention Duration: 2 lectures) ieee 
standards undergraduate eo 
without students 
heservauions Experiment 3: 1 college in Mobile Ongoing Course MOCA Response System Academic 
Experiments 39 the South ee a eee Not for Credit earns 
2and 3: undergraduate SOUS eat 7; ean a ae Fi gee 
Meets WWC seugoRie answere 0 questions or credit 
: outside of class time prior to the 
group design ' Ape 
standards with eee: 
reservations (Intervention Duration: 2 lectures) 
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Outcome 
Domain and WWC 
(ee: av ele-vom sive 


Tad (al oy al 3 Setting TaecyaUclalecelam@eorarel lacey) (@oynn) ex-Taexolam Gelato! i dco) s MS) 74-1 
Moreau 113 1 college Clickers: Students answered Business As Usual: Academic 
(2009) undergraduate (PA) 12-15 questions using clickers Comparison sections had achievement: 
RCT students during the first 30 minutes of the same lesson, examples, g = +0.38 
class, and were allowed to discuss practice problems, 
Meets WWC responses with their neighbors. homework, quizzes, and 
group design During the next 30 minutes, tests as intervention 
standards with new material was presented and __ sections, but they did not 
reservations students answered 2-3 questions have embedded questions 
using clickers to reinforce these and did not use clickers. 
concepts. 
(Intervention Duration: 1 
semester) 
Perez & 2H 1 college Socratic Agent (Microsoft's Text Feedback: Amessage Academic 
Solomon undergraduate (FL) Genie): Students received informed students that achievement: 
(2005) students feedback from an animated an incorrect choice had g = +0.03 
RCT genie when they made an been made but provided 
incorrect choice during a no hints about the correct 
Meets WWC computer-based simulation of an answer. 
group design electronics disassembly task. 
standards : ; 
tne: (Intervention Duration: 20-45 
reservations PIUNUNeS 
Yourstone, 190 1 college Clicker Quizzes: Students took 21 Paper and Pencil Quizzes: Academic 
Kraye, & undergraduate (NM) quizzes using clickers. Each quiz Students could answer the achievement? 
Albaum students included 7-12 questions. Foreach quiz questions in any order 
(2008) question, the instructor provided —_as long as they did not 
RCT aset amount of time, depending spend a total of more than 
on the type of question. At 15 minutes on the quiz. 
Meets WWC the end of allotted time for Quizzes were returned the 
group design each question, students were following week, and at that 
standards shown the number of students time students could ask 
without who selected each response about questions they got 
reservations option and the correct option. wrong. 
For questions in which many 
students answered incorrectly, 
the instructor explained the 
correct answer to the class. 
(Intervention Duration: 1 
semester) 
Domain Averages for Recommendation 4 (studies not testing ILO and OLI) Academic 
achievement 
g = +0.37* 
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Outcome 
Domain and WWC 
Calculated Effect 


Participants Setting aesyaUclalacelam@eodareliaceyay Comparison Condition Size 
Bowen, 605 6 public nteractive Learning Online (ILO): Face-to-Face Instruction: Academic 
Chingos, undergraduate universities Students assigned to the hybrid Students assigned to achievement: 
Lack, & students (NY and statistics courses participated in the traditional statistics @= +002 
Nygren MD) LO, which included machine- courses were taught 
(2014) guided instruction and face- introductory statistics asit Credit accumulation 
Additional to-face instruction each week. is usually offered at their and persistence: 
Source: These sections were delivered institutions with face-to- Gas, 19" 
in a hybrid mode, in which face instruction. 
Bowen, most of the instruction was 
Chingos, delivered through interactive 
Lack, & online materials, but the online 
Nygren instruction was supplemented 
(2012) by a weekly 1-hour face-to- 
eT face session, so that students 
could ask questions or be given 
Meets wwe targeted ete by es 
group design instructor. The ILO intervention 
standards included textual explanations of 
without concepts, worked examples, and 
HONEY practice problems. Students were 
also required to manipulate data 
using statistical software. 
(Intervention Duration: 1 
semester 
Evans, Yaron, 45 1 university OL! Stoichiometry Review Text-Only Study Guide: Academic 
& Leinhardt undergraduate (PA) Course: This course includes an Students assigned to the achievement: 
(2008) students overarching real-world story, an comparison group were G= 0636 
Additional explorato y virtual laboratory, a given a text-only study 
eauiee: variety of practice contexts, and guide. The study guide 
immediate feedback on actions covers the same content 
Evans (2007) and submitted responses. as the online review course 
RCT (Intervention Duration: 20-25 rien berraratn es 
Meets wwe nou) interface, o ae 
gioup design The review guide takes an 
standards estimated 12 to 15 hours 
ers to complete. 
reservations 
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Outcome 
Domain and WWC 
(eave leven sivas 


Participants Setting Taie=Yavcelaia cela Mm @elaveliace)a) Comparison Condition Size 

Lovett, 61 1 university Open Learning Initiative (OLN), Traditional Course: Academic 

Meyer, & undergraduate (PA) Accelerated Course: Students Students received achievement: 

Thille (2008) = students enrolled in an interactive online traditional instruction in g=+0'55* 

Additional statistics course. Students in a 15-week introductory 

Source: the OLI course acquired most statistics course. The 

of the course content online, content was the same 

Lovett, meeting with an instructor as that delivered in the 

Meyer, & approximately two times a intervention condition. 

Thille (2010) week for 50-minute sessions to 

RCT ask questions and review more 

challenging material. The course 

Meets WwWc was ate in an accelerated 

group design format (8 weeks instead of 15). 

standards 

without (Intervention Duration: 8 weeks) 

reservations 

Schunn & 188 Large, Open Learning Initiative (OLI), Traditional Course: Academic 

Patchan undergraduate public Logic and Proofs Course: Students received achievement: 

(2009) students research Students enrolled in an traditional instruction in a g=-0.10 

QED institution interactive online logic and 13-week logic and proofs 

in the US proofs course. This course course. The content was 

Meets WWC combined online instruction with the same as that delivered 

group design a small amount of targeted face- _in the intervention 

standards with to-face instruction. condition. 

reservations ; ; 

(Intervention Duration: 13 weeks) 

Domain Averages for Recommendation 4 (studies testing ILO and OLI) Academic 
achievement 
g=+0.11 
Credit 


accumulation and 
persistence: 
g = +0.19* 


@ Author reports findings as positive and statistically significant but effect sizes cannot be estimated from information provided in the study. The statistical 


significance of the findings could not be confirmed by WWC. 


* = statistically significant at the .05 level. 
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Recommendation 5. Use simulation technologies that help 
students engage in complex problem-solving. 


Rationale for Level of Evidence: Minimal 


WWC staff and the panel assigned a minimal level of evidence based on two studies that examined the effects 

of a web-based simulated problem-based learning experience (see Table C.5). One study meets WWC group 
design standards without reservations’ and one study meets WWC group design standards with reservations.'” 
The studies examined the effects of a 90-minute game-based immersive-world exercise''° and a semester-long 
community health simulation" on student academic achievement, as measured by a short content test and scores 
on a final exam, respectively. Results of a meta-analysis of these two studies indicate that web-based simulated 
problem-based learning experiences have a statistically significant positive impact on academic achievement 

(g =0.99, p < 01). 


The two studies that meet standards and that are used as evidence to support this recommendation involve the 
use of a computer simulation and games. The successful use of simulations and immersive technologies in medical 
education, health care delivery, and military and job training points to the promise of using technology to simulate 
complex problems that can be presented to and solved by students in other areas of postsecondary education." 
The minimal evidence rating assigned to this recommendation is due to the limited research evidence that meets 
WWC standards and the low level of external validity of that evidence. As technologies for simulating complex 
problems continue to develop and their applications for higher education expand, researchers should seek 
opportunities to study the impacts of these technologies on learning outcomes. 


This rating is supported by strength of the evidence according to the following criteria: 


e Consistency of Effects on Relevant Outcomes. The two studies used as evidence to support this 
recommendation found positive effects on measures of academic achievement. 


e Internal Validity of Supporting Evidence. The two studies were RCTs. One study had low attrition 
and meets WWC group design standards without reservations.'"’ The other study had high attrition and 
demonstrated baseline equivalence and meets WWC group design standards with reservations.'* 


e Relationship Between the Evidence and Recommendation 5. The two studies used as evidence to 
support this recommendation were direct tests of the recommendation. One study examined the impact of 
a web-based immersive-world game.'’? The other study examined the impact of a web-based community 
health simulation on academic achievement in the context of an upper-level undergraduate health behavior 
course.''® The panel agreed that the academic achievement measures were testing higher-order thinking skills 
and understanding of complex concepts. 


e External Validity of Supporting Evidence. The interventions occur in an upper-level undergraduate health 
behavior course at a public 4-year university in one study'’ and a laboratory setting in the other study.'' 
Though the intervention lasted the entire semester in the health behavior course, the intervention was 
conducted in a short, 90-minute lab setting in the other study. The sample size across the two studies was 
69 students. 
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Table C.5. Study providing evidence for Recommendation 5: Use simulation technologies that help 
students engage in complex problem-solving 


Participants 


Setting 


Outcome 
Domain and WWC 
Calculated Effect 


5 
undergraduate 
students 


1 large, 
Midwestern 
university 


Barab et al. 
(2008) 


RCT 


Meets WWC 
group design 
standards 
without 
reservations 


18 
undergraduate 
students 


1 college in 
the Western 
United 
States 


Spinello & 
Fischbach 
(2008) 


Additional 
source: 
Spinello 
(2004) 

RCT 

Meets WWC 
group design 
standards with 
reservations 


Domain Averages for Recommendation 5 


TalesvavcVala (elem @eyareliacoya Comparison Condition Size 

Immersive World Condition: Expository Text Condition: | Academic 

Students explored a multi-user Students were presented achievement: 

virtual environment called the the same information g=+1.07* 

Taiga Park on a computer, which in a 38-page electronic 

simulated an aquatic habitat. textbook on a website. 

The participants interacted with The contents were broken 

other characters within functional down into four separate 

groups in the game. In the game, instructional water 

participants were provided quality-based activities. 

information via a first-person After each section, the 

narrative and given options for participants were given 

their response. They also collected = the opportunity to review 

water samples and brought the contents and then 

them to a virtual laboratory for took the test. 

analysis. Participants took quizzes 

throughout their experience and 

complete three quests to report 

their findings in the exploration. 

Intervention Duration: 90 minutes) 

Web-Based Community Health Traditional Project-Based Academic 

Simulation: Students were Learning Assignment: achievement: 

presented with a web-based virtual Students were assigned g = +0.85* 

community and tools to intervene —_ to work with an actual 

during a simulated infectious community to identify 

disease outbreak. Components of and assess community 

the simulation include information — health needs and design 

resources, an interactive mapping and pilot a health behavior 

tool, an online survey builder that intervention. The semester 

allows students to design and timeline prevents students 

administer a survey and receive from observing health 

simulated results, a synchronous outcomes that may 

communication tool that allows have resulted from the 

students to conduct interviews intervention. 

with a virtual patient and physician, 

and a dynamic budgeting system. 

The web-based simulation 

platform allowed for an accelerated 

timeline for students to implement 

health interventions and monitor 

the resulting health outcomes over 

the course of a semester. 

(Intervention Duration: 1 semester) 
Academic 
achievement 
g=+0.99* 


* = statistically significant at the .05 level. 
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